Three-dimensional display device and three-dimensional display method

ABSTRACT

A three-dimensional display device configured to display a main image and an additional image on a screen includes a display region candidate decider that decides one candidate region from a plurality of region candidates for the additional image to be superimposed on the main image on the screen, a depth suitability determiner that determines whether a difference between a depth of the main image displayed at a boundary region and a depth of the additional image is within a predetermined tolerance range, and an image composer that, when the difference in depth between the depth of the main image displayed at the boundary region and the depth of the additional image is within the tolerance range, superimposes the additional image upon the main image at the candidate region, thereby composing a composite image of the main image and the additional image, and displays the composite image on the screen.

CROSS REFERENCE TO RELATED APPLICATION

This is a continuation application of U.S. patent application Ser. No.14/701,537, filed May 1, 2015, which is a Bypass Continuationapplication of International Application No. PCT/JP2014/003796, filed onJul. 17, 2014, and which claims the benefit of Japanese PatentApplication No. 2013-175071, filed Aug. 26, 2013, and JapaneseApplication No. 2013-175061, filed Aug. 26, 2013, all of the disclosuresof which are expressly incorporated herein by reference in theirentireties.

BACKGROUND 1. Technical Field

The present disclosure relates a three-dimensional (hereinafter “3D”)display device and a 3D display method.

2. Description of the Related Art

There are display devices that display moving images or still images,where additional information is superimposed on the image, to improveusability. Japanese Unexamined Patent Application Publication No.6-292240 discloses a method for displaying a depth-direction ruler onthe displayed image if the display device displays 3D images (e.g., aleft-eye image and right-eye image). This makes the depth-wise positionof an object to be more readily comprehended. Japanese Unexamined PatentApplication Publication (Translation of PCT Application) No. 2009-542362discloses a method for displaying information of forceps outside of aviewing angle, in an outer frame portion adjacent to the display image.Thus, the user can acquire information of the position and state offorceps outside of the display range. However, there has been recognizeda need for further improvement over the above-described related art.

SUMMARY

In one general aspect, the techniques disclosed here feature athree-dimensional display device that includes: a display regioncandidate decider that decides one candidate region from a plurality ofregion candidates for an additional image to be superimposed on a mainimage on a screen, a depth suitability determiner that determines, basedon a boundary region that is within a predetermined distance from aboundary line between the candidate region and outside of the candidateregion in the main image, whether a difference between a depth of themain image displayed at the boundary region and a depth of theadditional image is within a predetermined tolerance range, and an imagecomposer that, when it is determined that the difference in depthbetween the depth of the main image displayed at the boundary region andthe depth of the additional image is within the tolerance range,superimposes the additional image upon the main image at the candidateregion, thereby composing a composite image of the main image and theadditional image, and displays the composite image on the screen.

These general and specific aspects may be implemented using a system, amethod, and a computer program, and any combination of systems, methods,and computer programs.

Additional benefits and advantages of the disclosed embodiments willbecome apparent from the specification and drawings. The benefits and/oradvantages may be individually obtained by the various embodiments andfeatures of the specification and drawings, which need not all beprovided in order to obtain one or more of such benefits and/oradvantages.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram illustrating a functional configuration of a3D display device according to a first embodiment;

FIG. 2A is a diagram illustrating an example of additional imageposition information;

FIG. 2B is a schematic diagram illustrating an example of a coordinatessystem according to the first embodiment;

FIG. 3 is a diagram illustrating an example of size information of anadditional image;

FIG. 4 is a diagram illustrating an example of region information of anadditional image;

FIG. 5 is a flowchart illustrating processing operations of the 3Ddisplay device according to the first embodiment;

FIG. 6A is a schematic diagram illustrating an example of lines of viewin a case where there is no contradiction in depth between a main imageand additional image;

FIG. 6B is a schematic diagram illustrating an example of lines of viewin a case where there is contradiction in depth between a main image andadditional image;

FIG. 7 is a block diagram illustrating a functional configuration of a3D display device according to a second embodiment;

FIG. 8 is a flowchart illustrating processing operations of the 3Ddisplay device according to the second embodiment;

FIG. 9 is a block diagram illustrating a detailed functionalconfiguration of part of the 3D display device according to the secondembodiment;

FIG. 10A is a schematic diagram illustrating an example of a displayposition of an additional image according to the second embodiment;

FIG. 10B is a schematic diagram illustrating an example of a displayposition of an additional image according to the second embodiment;

FIG. 10C is a schematic diagram illustrating an example of a displayposition of an additional image according to the second embodiment;

FIG. 10D is a schematic diagram illustrating an example of a main imageaccording to the second embodiment;

FIG. 11A is a schematic diagram illustrating an example of an additionalimage and a boundary region according to the second embodiment;

FIG. 11B is a schematic diagram illustrating an example of a displayposition of an addition image for a left-eye image according to thesecond embodiment;

FIG. 11C is a schematic diagram illustrating an example of a right-eyeimage according to the second embodiment;

FIG. 12A schematic diagram illustrating an example of IDs given topartial regions of a boundary region of an additional image according tothe second embodiment;

FIG. 12B is a diagram illustrating an example of disparity informationat a boundary region according to the second embodiment;

FIG. 13A is a schematic diagram illustrating an example of a displayregion of an additional image, an additional image display-forbiddenregion, and a boundary region, according to the second embodiment;

FIG. 13B illustrates an example of information of difference in depthfor a display region of an additional image including information ofdistance from a display-forbidden candidate region according to thesecond embodiment.

FIG. 14A is a schematic diagram illustrating an example of a small frameof an additional image according to the second embodiment;

FIG. 14B is a schematic diagram illustrating an example of a large frameof an additional image according to the second embodiment;

FIG. 15 is a block diagram illustrating a functional configuration of a3D display device according to a first modification of the secondembodiment;

FIG. 16 is an example of display-forbidden region information accordingto a first modification of the second embodiment;

FIG. 17 is a block diagram illustrating a detailed functionalconfiguration of a part of the 3D display device according to the firstmodification of the second embodiment;

FIG. 18 is a block diagram illustrating a functional configuration of a3D display device according to a second modification of the secondembodiment;

FIG. 19 is a block diagram illustrating a detailed functionalconfiguration of a part of the 3D display device according to the secondmodification of the second embodiment;

FIG. 20A is a schematic diagram illustrating an example of a boundaryplane of a display region candidate for an additional image, a boundaryregion, and a boundary adjacent region, according to the secondmodification of the second embodiment;

FIG. 20B illustrates an example of IDs given to partial regions of aboundary region and a boundary plane adjacent region according to thesecond embodiment;

FIG. 21 is an example of disparity information of boundary regions andboundary plane adjacent regions according to the second modification ofthe second embodiment;

FIG. 22 is a block diagram illustrating a detailed functionalconfiguration of a part of a 3D display device according to the secondmodification of the second embodiment;

FIG. 23 is a block diagram illustrating a detailed functionalconfiguration of a part of the 3D display device according to a thirdmodification of the second embodiment;

FIG. 24 is a block diagram illustrating a detailed functionalconfiguration of a part of the 3D display device according to the fourthmodification of the second embodiment;

FIG. 25 is a flowchart illustrating detailed operations of part ofprocessing operations of the 3D display device according to the fourthmodification of the second embodiment;

FIG. 26 is a flowchart illustrating processing operations of a 3Ddisplay device according to a fifth modification of the secondembodiment;

FIG. 27A is a schematic diagram illustrating an example of an input unitaccording to the fifth modification of the second embodiment;

FIG. 27B is a schematic diagram illustrating an example of a display fordescribing the input unit according to the fifth modification of thesecond embodiment;

FIG. 28A is a schematic diagram illustrating an example of a display fordescribing input of additional image size according to a sixthmodification of the second embodiment;

FIG. 28B is a schematic diagram illustrating an example of a display fordescribing input of additional image size according to the sixthmodification of the second embodiment;

FIG. 29 is a block diagram illustrating a functional configuration of a3D display device according to a seventh modification of the secondembodiment;

FIG. 30 is a schematic diagram illustrating an example of storedcontents, stored in an additional image region storage unit;

FIG. 31 is a flowchart illustrating processing operations of the 3Ddisplay device according to the seventh modification of the secondembodiment;

FIG. 32A is a flowchart illustrating processing operations of a 3Ddisplay device according to an eighth modification of the secondembodiment;

FIG. 32B is a flowchart illustrating processing operations of the 3Ddisplay device according to the eighth modification of the secondembodiment;

FIG. 33 is a block diagram illustrating a detailed functionalconfiguration of a display region candidate deciding unit according tothe eighth modification of the second embodiment;

FIG. 34 is a flowchart illustrating processing operations of the displayregion candidate deciding unit according to the eighth modification ofthe second embodiment;

FIG. 35 is a block diagram illustrating a functional configuration of a3D display device according to a ninth modification of the secondembodiment;

FIG. 36 is a flowchart illustrating processing operations of the 3Ddisplay device according to the ninth modification of the secondembodiment;

FIG. 37 is a block diagram illustrating a detailed functionalconfiguration of a display region candidate deciding unit according tothe ninth modification of the second embodiment;

FIG. 38A is a schematic diagram illustrating an example of a layout ofan additional image according to the ninth modification of the secondembodiment;

FIG. 38B is a schematic diagram illustrating an example of a layout ofan additional image according to the ninth modification of the secondembodiment;

FIG. 38C is a schematic diagram illustrating an example of a layout ofan additional image according to the ninth modification of the secondembodiment;

FIG. 39 is a flowchart illustrating processing operations of the displayregion candidate deciding unit according to the ninth modification ofthe second embodiment;

FIG. 40A is a schematic diagram illustrating an example of a case ofdisplaying all display region candidates for additional images accordingto the fifth through ninth modifications of the second embodiment on adisplay;

FIG. 40B is a schematic diagram illustrating an example of and operatinginput unit for displaying all display region candidates for additionalimages, according to the fifth through ninth modifications of the secondembodiment, on a display, and selecting one additional image displayregion candidate that is displayed;

FIG. 41 is a block diagram illustrating a functional configuration of a3D display device according to a third embodiment;

FIG. 42 is a diagram illustrating an example of depth differenceinformation for each display region of additional images according tothe third embodiment;

FIG. 43 is a flowchart illustrating partial operations of processingoperations of the 3D display device according to the third embodiment;

FIG. 44 is a flowchart illustrating detailed partial operations ofprocessing operations of the 3D display device according to the thirdembodiment;

FIG. 45 is a flowchart illustrating detailed partial operations ofprocessing operations of the 3D display device according to the thirdembodiment;

FIG. 46 is a block diagram illustrating a functional configuration of a3D display device according to a fourth embodiment; and

FIG. 47 is a flowchart illustrating processing operations of the 3Ddisplay device according to the fourth embodiment.

DETAILED DESCRIPTION

Underlying Knowledge Forming Basis of the Present Disclosure

First, description will be given regarding studies made by the presentinventors to reach the aspects of the present disclosure. The presentinventors have found that the following problems occur in the relatedart.

In a case of displaying a new object in part of a 3D image, like theruler in Japanese Unexamined Patent Application Publication No.6-292240, there are cases where the difference between the displayeddepth position of the object decided by the disparity of the position ofthe object in the left-eye image and the position of the object in theright-eye image, and the depth position of an object displayed in theoriginal 3D image is too great, or where there is contradiction in thedepth position (an object at a deeper position hides an nearer object).The display in the outer frame portion in Japanese Unexamined PatentApplication Publication (Translation of PCT Application) No. 2009-542362has also had a similar problem in that there are cases wherecontradiction occurs where the difference between a displayed image inthe screen and a displayed image in the frame is too great, or wherethere is contradiction in the depth position between the displayed imagein the screen and a displayed image in the frame. Such excessively greatdifference in depth and contradiction in depth positions cause userdiscomfort and fatigue, placing a large load on the user. The presentinventors studied the above problems to reach the aspects of the presentdisclosure.

According to one aspect of the present disclosure, a three-dimensionaldisplay device includes: a display region candidate deciding unit thatdecides one candidate region from a plurality of display regioncandidates of an additional image which shields part of a main image ofa three-dimensional image on a screen; a depth suitability determinationunit that determines, in a case of assuming that the additional image isto be displayed in the candidate region which the display regioncandidate deciding unit has decided, whether or not a difference indepth between depth of the main image displayed at a boundary regionwhich is a region on the main image and within a predetermined distancefrom a boundary line of the candidate region, and the depth of theadditional image, is within a predetermined tolerance range; an imagecompositing unit that, in a case where determination is made by thedepth suitability determination unit that the difference in depth iswithin the tolerance range, superimposes the additional image upon thecandidate region on the main image, thereby compositing the main imageand the additional image, and displays an image obtained as a result ofthe compositing on the screen; and a possibly-unsuitable region decidingunit that decides, in the main image, a first region that has apossibility of the depth protruding to a near side beyond apredetermined depth range, and a second region that has a possibility ofthe depth recessing to a far side beyond a predetermined depth range.The display region candidate deciding unit further decides a candidateregion to shield the first region and the second region decided by thepossibly-unsuitable region deciding unit.

According to this aspect, an additional image is displayed in a casewhere the difference in depth between the main image and additionalimage across the boundary line of the additional image is within thetolerance range. Depth contradiction due to displaying the additionalimage at a region greatly protruding to the near side, for example, canbe resolved. Accordingly, user discomfort and fatigue can be prevented.

Embodiments will be described in detail with reference to the drawings.Note that the descriptions made in the embodiment described below areall either comprehensive or specific examples. Values, shapes,materials, components, placements and connections of components, steps,orders of steps, and so forth, set forth in the following embodiments,are only exemplary, and are not intended to restrict the Claims.Components in the following embodiments which are not included in anindependent Claim indicating the highest concept are described as beingoptional components.

Description in detail beyond what is necessary may be omitted in thefollowing. For example, detailed description of well-known items andrepetitive description of configurations which are essentially the samemay be omitted. This is to avoid unnecessarily redundant description,and to facilitate understanding of those skilled in the art.

First Embodiment

Configuration

FIG. 1 is a block diagram illustrating a functional configuration of a3D display device 10 according to a first embodiment. The 3D displaydevice 10 displays 3D images. That is to say, the 3D display device 10performs stereoscopic display of images. More specifically, the 3Ddisplay device 10 displays 3D images by a head mounted display (HMD). AnHMD displays 3D images by a display in close proximity the eyes of theuser. An HMD uses a left-eye image and a right-eye image which havedisparity, displaying the right-eye image on a display in closeproximity with the right eye of the user, and displaying the left-eyeimage on a display in close proximity with the left eye of the user. Theleft and right displays are situated so that the left eye cannot see theright-eye image, and the right eye cannot see the left-eye image. Thereis another type of stereoscopic display, called 3D glasses, examples ofwhich include liquid crystal shutter glasses and polarized glasses. Theuser wearing the 3D glasses is presented with a left-eye image and aright-eye image which have disparity. The 3D display device 10 may alsodisplay 3D images by a naked-eye stereoscopic system. A naked-eyestereoscopic system is a stereoscopic system which does not use 3Dglasses, examples of which include the parallax barrier display andlenticular lens display.

The 3D display device 10 illustrated in FIG. 1 includes a main imageacquisition unit 110, an additional information acquisition unit 120, adepth information generating unit 130, an additional image positionstorage unit 140, an additional image size storage unit 150, a displayregion candidate deciding unit 160, a depth suitability determinationunit 170, an image compositing unit 180, and a display 190.

The main image acquisition unit 110 acquires image information for 3Ddisplay having left and right disparity (3D images). The images havebeen imaged at the same time by two cameras arrayed in tandem, forexample. The image of one camera is the right-eye image, and the imageof the other camera is the left-eye image. The images may be currentlybeing imaged, or may be recorded beforehand. Alternatively, the 3Ddisplay images having left and right disparity may be an image imaged bya single camera which has been subjected to image processing so as togenerate left and right images by disparity.

The additional information acquisition unit 120 acquires additionalinformation to be added to the image information which the main imageacquisition unit 110 acquires. An example of additional information isinformation temporally synchronous with the image information, that hasbeen acquired by a sensor other than the camera but situated nearby thecamera imaging the main image. Another example of additional informationis a 3D image created separately from the main image.

The depth information generating unit 130 obtains the left and rightdisparity of the 3D image which the main image acquisition unit 110 hasobtained, and generates depth information of the main image. Forexample, the depth information generating unit 130 extracts correlationpoints between the right-eye image and the left-eye image, obtains theleft and right disparity of the correlation points, and calculates thedepth.

The additional image position storage unit 140 stores, for theadditional information acquired by the additional informationacquisition unit 120, the position for display on the screen of thedisplay 190. For example, the additional image position storage unit 140stores coordinates in 3D space when the 3D image is displayed, as theposition. Alternatively, the additional image position storage unit 140may store the display position on the display 190 of each of theleft-eye image and right-eye image as the position. FIG. 2A illustratesan example of additional image position information stored in theadditional image position storage unit 140. FIG. 2B schematicallyillustrates an example of 3D coordinates for identifying additionalimage positions. The additional image position storage unit 140 storesan additional image position ID for identifying the position ofadditional images (images representing additional information), and thecenter-of-gravity position of the additional images corresponding to theadditional image position IDs, as illustrated in FIG. 2A. Thecenter-of-gravity position is a point on coordinate axes arranged asillustrated in FIG. 2B which represents 3D space.

The additional image size storage unit 150 stores the size of displayingthe additional information acquired by the additional informationacquisition unit 120 as an additional image on the screen of the display190. FIG. 3 illustrates an example of size information of additionalimages stored in the additional image size storage unit 150. Theadditional image size storage unit 150 in the example in FIG. 3 storesadditional image size IDs to identify size information of additionalimages, and the dimensions of the additional images in terms of heightand width. In the first embodiment, the additional images are shaped asrectangles, with the size thereof specified by the length of the sidesin the height direction and the length of the sides in the widthdirection. The way in which the sizes of the additional images areexpressed differs depending on the shape. For example, if an additionalimage is an ellipse, the size is specified by the major and minor axes.

While description has been made that the additional image positionstorage unit 140 and the additional image size storage unit 150respectively store the position and size of additional images, theposition and size of additional images may be stored in a single storageunit, such as an additional image region storage unit for example. In acase where the additional image is a polygon for example, the additionalimage region storage unit stores the coordinate positions in 3D spacefor the vertices of the polygon. This enables the position and size tobe stored together. FIG. 4 illustrates an example of contents stored inan additional image region storage unit in a case of storing theposition and size of additional image as a region. In the exampleillustrated in FIG. 4, the additional images are rectangles which havefour sides, parallel to the four sides of the screen of the display 190,and the coordinate positions of the four vertices are storedcorresponding to IDs identifying the regions of the additional images.

The display region candidate deciding unit 160 decides candidates for adisplay region to display one or a plurality of additional informationon the screen of the display 190 as the additional image. In thefollowing description, the term “display region” as such means a displayregion for additional information or an additional image.

The depth suitability determination unit 170 detects depth differencegreater than a predetermined value, or depth contradiction, in thedifference in depth between the display region of an additional imageand a peripheral portion of the display region of an additional image ina main image generated by the depth information generating unit 130.This detection is made based on information of additional image displayregion candidates decided by the display region candidate deciding unit160 and depth information of the main image generated at the depthinformation generating unit 130.

The image compositing unit 180 displays the 3D image acquired by themain image acquisition unit 110 on the display 190, and composites themain image and an additional image to display the additional informationacquired by the additional information acquisition unit 120 in at leastone of the candidates for a display region which the display regioncandidate deciding unit 160 has decided, as an additional image.

The display 190 displays the image composited by the image compositingunit 180 on the screen.

Operations

FIG. 5 is a flowchart illustrating processing operations of the 3Ddisplay device 10 according to the first embodiment. First, the mainimage acquisition unit 110 acquires image information for 3D displaythat has left and right disparity, and the additional informationacquisition unit 120 acquires additional information corresponding tothe main image indicated by the image information (step S1100). Theadditional information may be information temporally synchronous withthe main image, or information not temporally synchronous.

Next, the display region candidate deciding unit 160 decides candidatesfor the display region, based on the size of the additional image storedin the additional image size storage unit 150 and the position of theadditional image stored in the additional image position storage unit140 (step S1200). One or a plurality of information of additional imagesize and position is stored, and the display region candidate decidingunit 160 decides one or more additional image display region candidates.The display region candidate deciding method is as follows, for example.First, one additional image position information stored in theadditional image position storage unit 140 is selected, thereby decidinga display position, and further, one additional image size informationstored in the additional image size storage unit 150 is selected,thereby deciding the additional image size.

The depth suitability determination unit 170 extracts a boundary line orboundary plane of the display region candidate of the additional imagedecided in step S1200. A boundary plane is a depth-direction faceorthogonal to the plane of the display 190. The depth suitabilitydetermination unit 170 identifies a portion in the main image acquiredin step S1100 in contact with the boundary line or boundary plane of theadditional image, and extracts depth information (step S1300). The depthinformation generating unit 130 generates and holds the depthinformation of the main image by the time that the main imageacquisition unit 110 acquires the main image in step S1100 and the depthsuitability determination unit 170 extracts the depth information instep S1300.

Further, the depth suitability determination unit 170 compares the depthof the display region of the additional image obtained by information ofthe position of the additional image stored in the additional imageposition storage unit 140 by the display region candidate deciding unit160, with the depth of the main image at a portion in contact with theboundary line or boundary plane of the additional image extracted instep S1300 (step S1400).

The depth suitability determination unit 170 determines whether or notthe difference in depth between the main image and additional imagedisplayed across the boundary lines or boundary plane exceeds apredetermined tolerance range (step S1500). The difference in depth isexpressed on a depth-wise coordinate axis, on which the value increasestoward the protruding side for example, which is to say toward the userfrom the display 190. In this case, if a value obtained by subtractingthe depth value of the main image nearby the boundary line from thedepth value of the additional image is a positive value, this means thatthe additional image is nearer, protruding toward the user, and the mainimage is at a deeper position than the additional image. If a valueobtained by subtracting the depth value of the main image nearby theboundary line from the depth value of the additional image is a negativevalue, this means that the additional image is of a smaller value andthe main image is at a position protruding toward the user. Theadditional image is displayed hiding a portion of the main image, so thehidden main image is at a deeper position, and the hiding additionalimage is nearer than the main image, so that the additional imageoverwrites the main image. However, if this value is a negative value,this indicates that a depth contraction has occurred in which the depthposition of the additional image which has been set is deeper than thatof the main image. Depth contradiction causes user fatigue and 3Dsickness, so a tolerance range is set where, for example, depthcontradiction exceeding 1 cm is unacceptable.

On the other hand, even if there is no depth contradiction, it is knownthat great depth differences at close portions lead to user fatigue.Accordingly, so a tolerance range is set where, for example, depthdifference exceeding 15 cm is unacceptable. Thus, the tolerance range is−1 cm to 15 cm, for example. In a case where determination is made instep S1500 that the difference in depth exceeds the tolerance range,i.e., step S1500 yields a result of “yes”, the flow advances to stepS1600. On the other hand, in a case where determination is made in stepS1500 that the difference in depth is within the tolerance range, i.e.,step S1500 yields a result of “no”, the flow advances to step S1700.

While the tolerance range for difference in depth will be described asbeing a fixed value of −0 cm to 15 cm here, the tolerance range may beset variably depending on the depth position of the additional image.The settings may be such that greater the depth of the additional image,i.e., the greater the degree of protrusion of the additional imagetoward the user, the narrower the tolerance range is, and the smallerthe depth coordinates of the additional image is and the farther theadditional image appears from the user, the larger the tolerance rangefor difference in depth is.

The display region candidate deciding unit 160 changes the displayposition of the additional image (step S1600). Changing of the displayposition is performed by selecting, from display positions stored in theadditional image position storage unit 140, a display position notselected as a display position candidate in step S1200. After stepS1600, the flow returns to step S1200.

The image compositing unit 180 composites the main image acquired instep S1100, and the additional image representing the additionalinformation acquired in S1100 (step S1700). More specifically, the imagecompositing unit 180 displays the main image on the screen of thedisplay 190, and superimposes the additional image over the main imageso as to be displayed on the display region selected from the displayregion candidates decided in step S1200 as the display region, therebycompositing the images.

The display 190 displays the 3D image composited in step S1700 (S1800).After displaying the 3D image on the display 190 in step S1800, the flowreturns to step S1100. Repeating steps S1100 through S1800 causes the 3Ddisplay device 10 to acquire images and additional information inincrements of processing, and continue displaying of images. The presentembodiment is applicable in cases where the contents of the main imageand additional image are moving images, as well.

Advantages and Effects

Thus, the 3D display device 10 according to the present embodimentdecides display regions for additional images when displaying additionalinformation (additional images) over or adjacent to 3D images, such thatstates where there is a great difference in depth without and outside,or states where there is depth contradiction are avoided. Accordingly,user discomfort and fatigue due to excessively large difference in depthor depth contradiction can be prevented. While the first embodiment hasbeen described with regard to an arrangement where the tolerance rangeis set for difference in depth, the tolerance range may be set fordifference in disparity as well.

FIGS. 6A and 6B schematically illustrate the relationship between thedepth position and disparity of the main image and additional imagedisplayed on the display 190. FIG. 6A illustrates a state where there isno depth contradiction, and the additional image is displayed on thenear side from the main image. FIG. 6B illustrates a state where thereis depth contradiction, and the additional image is displayed at adeeper position than the main image but is displayed without beinghidden by the nearer main image.

The coordinates on the display are set such that the values are largertoward the right and smaller toward the left, when facing the displayfrom the front. In a case of calculating disparity as a value obtainedby subtracting the value of the x-coordinate of the left-eye image fromthe value of the x-coordinate of the right-eye image, the disparity of apoint displayed on the plane of the display is 0, and the disparity of apoint viewed as appearing deeper than the plane of the display,indicated by the left-facing arrows in FIGS. 6A and 6B, is a positivevalue. On the other hand, the disparity of a point viewed as appearingnearer than the plane of the display, indicated by the right-facingarrows in FIGS. 6A and 6B, is a negative value.

In such a relationship, in a case where the value obtained bysubtracting the disparity of the outside of the display region of theadditional image from the disparity of the inside of the display regionof the additional image is negative, the depth of the inside of thedisplay region of the additional image is nearer and the outside of thedisplay region of the additional image is deeper. That is to say, thedisplay region of the additional image is nearer than the outside of thedisplay region, so the additional image is presented to the near side ofthe main image acquired by the main image acquisition unit 110, so thereis no depth contradiction.

On the other hand, in a case where the value obtained by subtracting thedisparity of the outside of the display region of the additional imagefrom the disparity of the inside of the display region is positive, asin the case in FIG. 6B, the depth of the inside of the display region isdeeper than the main image acquired by the main image acquisition unit110 and the additional image should be presented at a depth positionhidden by the main image, so there is depth contradiction. Accordingly,the tolerance range is set such that the tolerance range is small in acase where the value obtained by subtracting the disparity of theoutside of the display region from the disparity of the inside of thedisplay region is positive, and in a case where the value obtained bythe subtraction is a negative value, the tolerance range is set so thatthe value obtained by the subtraction is not smaller than a certainvalue. In a case where the value obtained by subtracting the disparityof the outside of the display region from the disparity of the inside ofthe display region is negative, a smaller value obtained by subtractingthe disparity of the outside of the display region from the disparity ofthe inside of the display region means a larger absolute value, whichmeans a greater difference in disparity, and thus greater difference indepth. The tolerance range in depth can thus be set by setting thedisparity, with the range of difference in disparity being stipulatedsuch that the difference in depth is no greater than a certain level.

While description has been made in the first embodiment where the depthsuitability determination unit 170 makes a binary determinationregarding whether or not the difference in depth between the main imageand additional image across the boundary plane of the additional imageexceeds the tolerance range, an arrangement may be made where asuitability is calculated from the magnitude of difference from thetolerance range that the difference in depth of the main image andadditional image has, and adjusts the display region of the additionalimage based according to this suitability. This suitability is expressedas a function representing the relationship between the difference indepth of the main image and additional image and suitability, or acorrelation table or the like.

Second Embodiment

While the 3D image (main image) and additional image were not describedas being an image and information of any particular type in the firstembodiment, the 3D image may be an endoscopic surgery image, and theadditional information represented by the additional image may includevital signs information during the surgery, such as pulse, bloodpressure, and so forth. The additional image may further include imageinformation such as a magnetic resonance imaging (MRI) image or computedtomography (CT) image taken before the surgery.

The 3D image in a second embodiment is a real-time image of an affectedarea being treated by endoscopic surgery using a stereo camera, and theadditional information is vital signs information of the patient duringthe surgery, and MRI image information taken and stored before thesurgery. The types of additional information are only exemplary, andother types of information may be displayed as additional information aswell.

Configuration

FIG. 7 is a block diagram illustrating a functional configuration of a3D display device 20 according to the second embodiment. Theconfiguration is the same as the 3D display device 10 according to thefirst embodiment illustrated in FIG. 1, other than the points that themain image acquisition unit 110 has been replaced by an endoscope camera111, the additional information acquisition unit 120 has been replacedby a vital signs sensor 121, and an MRI image storage unit 122, 3D imagecompositing unit 123, display-forbidden region storage unit 210, andinput unit 200 have been newly added. Portions which are the same asthose in FIG. 1 are denoted by the same reference numerals, anddescription will be omitted.

The 3D display device 20 includes the endoscope camera 111, vital signssensor 121, MRI image storage unit 122, 3D image compositing unit 123,depth information generating unit 130, additional image position storageunit 140, additional image size storage unit 150, display regioncandidate deciding unit 160, depth suitability determination unit 170,image compositing unit 180, display 190, display-forbidden regionstorage unit 210, and input unit 200.

The endoscope camera 111 is a 3D imaging endoscope camera including astereo camera.

The vital signs sensor 121 is a sensor attached to the body of thepatient during surgery. Examples of the vital signs sensor 121 include athermometer, an electrocardiogram, a sphygmomanometer, a pulse oximeter,an electroencephalograph, and so forth. In the present embodiment,description will be made with the vital signs sensor 121 serving as anelectrocardiogram and a sphygmomanometer.

The MRI image storage unit 122 stores 3D image information includingimages of the affected area to be treated by surgery, which have beenrecorded by an MRI system before surgery. While description is made inthe present embodiment regarding image information recorded by an MRIsystem before surgery as image information, image information of X-rayimages, CT image information, and so forth, may be used as well.

The 3D image compositing unit 123 composites image information stored inthe MRI image storage unit 122 into images of a format which can bedisplayed on the display 190, as specified slices or a spectrogram of aspecified range.

The display-forbidden region storage unit 210 stores informationrepresenting a region determined beforehand where an affected area to besubjected to surgery has been photographed, and information representinga display-forbidden region for the additional image. The user, who is asurgeon, needs to watch the affected area throughout the surgery.Accordingly, the additional image should be displayed outside of aregion displaying the affected area.

The display-forbidden region for the additional image is a rectangularfixed region centered on the screen center of the display 190, and thedisplay-forbidden region storage unit 210 stores coordinate positions ofthe four vertices of the rectangle as information representing thedisplay-forbidden region for the additional image in the secondembodiment. The endoscope camera 111 is situated by a guide pipe at alocation where the affected area to be treated by surgery can be readilyobserved. The endoscope camera 111 can zoom in toward the affected areaand zoom out away from the affected area, along the axial direction ofthe guide pipe. The guide pipe is fixed during surgery, so the endoscopecamera 111 does not move in any other direction than along the axis ofthe guide pipe.

While the display-forbidden region has been described as being a fixedregion in the second embodiment, the region may be changed depending onthe distance between the camera and the affected area. In this case, thedisplay-forbidden region storage unit 210 stores informationrepresenting the display-forbidden region corresponding to the distancebetween the camera and the affected area. For example, thedisplay-forbidden region storage unit 210 stores the coordinates of thevertices of the display region for each range of distance between thecamera and the affected area.

The input unit 200 is a processing unit where the user instructs whetheror not to display an additional image, and inputs conditions for thecontent to be displayed as an additional image. The user who is thesurgeon can display information acquired by the vital signs sensor 121and MRI images as necessary, by inputting instructions using the inputunit 200 during surgery. If the additional image is unnecessary, displayof the additional image can be cancelled. Further, the user who is thesurgeon can conduct conditions, such as whether an MRI image should bedisplayed as a slice 2D image or as a 3D image, what range should bedisplayed, and so forth, using the input unit 200.

Note that the additional information in the second embodiment is thevital signs information acquired by the vital signs sensor 121 duringsurgery, and the MRI image information stored before surgery. The twotypes of additional information are to be displayed as separateadditional images in respective display regions.

While description has been made in the second embodiment that vitalsigns information and MRI image information as to be displayed asseparate additional image, one additional image may be generated anddisplayed including multiple types of additional information together.

Operations

FIG. 8 is a flowchart illustrating processing operations of the 3Ddisplay device 20 according to the second embodiment. This flowchart isthe same as that in FIG. 5 in the first embodiment, other than stepsS1110 and S1210 have been added. Portions which are the same as those inFIG. 5 are denoted by the same reference numerals, and description willbe omitted.

First, the endoscope camera 111 acquires image information for 3Ddisplay having left and right disparity, as a main image, and the vitalsigns sensor 121 measures the current cardioelectric potential and bloodpressure of the patient as additional information (step S1100).

The input unit 200 acquires display control signals according to userinput, and detects instruction input to display an additional image(step S1110). In a case where additional image display instruction inputis detected in step S1110, i.e., in a case where step S1110 yields“yes”, the flow advances to step S1200. In a case where additional imagedisplay instruction input is not detected in step S1110, i.e., in a casewhere step S1110 yields “no”, the flow advances to step S1700.

In step S1700, the image compositing unit 180 composites an endoscopecamera image which is a main image with no additional image as a displayimage.

The display region candidate deciding unit 160 decides candidates forthe display region, from the size of the additional image stored in theadditional image size storage unit 150, and the position of theadditional image stored in the additional image position storage unit140 (step S1200). For example, the display region candidate decidingunit 160 arbitrarily selects one unselected combination of additionalimage size stored in the additional image size storage unit 150 andadditional image position stored in the additional image positionstorage unit 140, and decides a region represented by the selectedcombination as a display region candidate.

The display region candidate deciding unit 160 determines whether or notthe display region candidate decided in step S1200 includes thedisplay-forbidden region stored in the display-forbidden region storageunit 210 (step S1210). In a case where determination is made in stepS1210 that the display region candidate includes the display-forbiddenregion, i.e., in a case where step S1210 yields “yes”, the flow advancesto step S1600.

In step S1600, the display region candidate deciding unit 160 moves thedisplay position of the additional image and stores the moved displayposition in the additional image position storage unit 140. That is tosay, the display region candidate deciding unit 160 may move the displayposition of the additional image to a position where the display regioncandidate does not include the display-forbidden region, or may move thedisplay position of the additional image in a predetermined direction bya predetermined amount.

In a case where determination is made in step S1210 that the displayregion candidate does not include the display-forbidden region, i.e., ina case where step S1210 yields “no”, the flow advances to step S1300.

Repeating steps S1200, S1210, and S1600 decides the display regioncandidate while avoiding the display-forbidden region. The depthsuitability determination unit 170 extracts a boundary line or boundaryplane of the additional image display region candidate decided in stepS1200, and extracts depth information in the periphery of the boundaryline or boundary plane (step S1300).

The depth suitability determination unit 170 further compares the depthof the display region of the additional image which the display regioncandidate deciding unit 160 has obtained from the position informationof the additional image stored in the additional image position storageunit 140 with the depth of the main image at the portion adjacent to theboundary line or boundary plane of the additional image extracted instep S1300 (step S1400).

The depth suitability determination unit 170 determines whether or notthe difference in depth of the main image and additional image displayedacross the boundary line or boundary plane exceeds the predeterminedtolerance range (step S1500). The tolerance range is −1 cm to 15 cm, inthe same way as the first embodiment, for example. In a case wheredetermination is made in step S1500 that the difference in depth exceedsthe tolerance range, i.e., in a case where step S1500 yields “yes”, theflow advances to step S1600. In a case where determination is made instep S1500 that the difference in depth is within the tolerance range,i.e., in a case where step S1500 yields “no”, the flow advances to stepS1700.

In step S1600 the display region candidate deciding unit 160 moves thedisplay position of the additional image, stores the moved displayposition in the additional image position storage unit 140, andthereafter the flow returns to step S1200.

In step S1700, the image compositing unit 180 composites the main imageacquired in step S1100 and the additional image representing theadditional information acquired in step S1100. Specifically, the imagecompositing unit 180 displays the main image on the screen of thedisplay 190, and displays the additional image on the main image so asto be displayed in the display region which is the display regioncandidate decided in step S1200 as the display region, thus compositingthe image.

The display 190 displays the 3D image composited in step S1700 (stepS1800), and after displaying the 3D image on the display 190 in stepS1800, the flow returns to step S1100. Repeating steps S1100 throughS1800 causes the 3D display device 20 to acquire main images andadditional information in increments of processing, and continuedisplaying of 3D images. The present embodiment is applicable in caseswhere the contents of the main image and additional image are movingimages, as well.

Depth Suitability Determination Unit 170: Details of Steps S1300 toS1500

An example where there are two additional images will be described inthe second embodiment. Note that the number of additional images mayalso be one, or three or more. One of the additional images in thesecond embodiment is a graph showing the cardioelectric potential andblood pressure measured by the vital signs sensor 121. This will bereferred to as a “vital signs information image” here. In the graph ofthe vital signs information image, the horizontal axis is the passage oftime, and the vertical axis is hectopascals (HPa) which is a displayunit of blood pressure, and also microvolts (μV) which is a display unitof cardioelectric potential. The graph is composited as atwo-dimensional image. The other additional image is MRI imageinformation recorded before the surgery, and is a 3D image generated bycomputer graphics (CG, the image is hereinafter referred to as a “CGimage”). The MRI image storage unit 122 stores image information for CGimages including 3D information such as polygon data, for example.

The 3D image compositing unit 123 decides a range for display of the CGimage and a viewpoint of the CG image, based on information recorded inthe MRI image storage unit 122, and thus can generate a 3D CG image fromthis viewpoint. In the second embodiment, the vital signs informationimage is displayed as a planar image situated at a position floatingabove the display plane by 2 cm. The CG image is displayed at a depthposition where the display plane is the center-of-gravity of a 3Dobject. The images are generated such that the depth distance of 3Dobjects generated by CG is constant. For example, the depth distance isa position 3 cm on the far side of the display plane through a position3 cm on the near side of the display plate, for a total of 6 cm.

FIG. 9 is a block diagram illustrating a detailed partial configurationof the 3D display device 20 according to the second embodiment. Thedepth suitability determination unit 170 includes a boundary extractingunit 171, a periphery depth extracting unit 172, an additional imagedepth storage unit 173, a depth difference calculating unit 174, areference storage unit 175, and a determination unit 176.

The boundary extracting unit 171 obtains the boundary plane between themain image and the display region of each additional image, based on thedisplay region of additional images which the display region candidatedeciding unit 160 has decided in step S1200 and depth information of theadditional images stored in the additional image depth storage unit 173.For example, in a case where the four sides of a rectangular additionalimage are parallel to the four sides of the display, the boundary planeis obtained as an x-z plane and a y-z plane on the coordinates systemillustrated in FIG. 2B. In a case where the additional image is notrectangular with four sides parallel to the four sides of the display,the boundary plane is a plane parallel to the z-axis.

The periphery depth extracting unit 172 extracts depth information ofthe main image at the periphery of the boundary plane, from the depthinformation of the main generated by the depth information generatingunit 130, and the information of the boundary plane which the boundaryextracting unit 171 has obtained. Further, the periphery depthextracting unit 172 extracts depth information of the additional imagesat the periphery of the boundary planes from the depth information ofeach additional image stored in the additional image storage unit 173.

The depth difference calculating unit 174 calculates the differencebetween the depth of the main image and the depth of the additionalimage at each boundary plane, with regard to the depth of the main imageand the depth of the additional images at the periphery of the boundaryplane of the additional image regions extracted by the periphery depthextracting unit 172. The difference in depth is calculated as follows,for example. With regard to each of the main image side and additionalimage side across a single boundary plane, the depth differencecalculating unit 174 obtains one or more contact positions between theboundary plane and an in-image object, or a plane of an image in a caseof a 2D image. The depth difference calculating unit 174 extracts thez-axial coordinate value for each of the main image side and theadditional image side at the contact position. The depth differencecalculating unit 174 extracts the largest value of the z-axialcoordinate value at the contact position for each of the main image sideand additional image side. The depth difference calculating unit 174then calculates a value obtained by subtracting the largest value at themain image side from the largest value at the additional image side asthe difference in depth. In a case where each region has multipleboundary planes, the depth difference calculating unit 174 outputs thelargest value in difference in depth to the determination unit 176 asthe difference in depth at the region.

While the difference in depth has been decided here based on the largestvalue of z-axial coordinate values of the contact position with theboundary plane, other values may be used, such as the average value ofz-axial coordinate values, or the like. While the difference in depthhas been obtained here from coordinate values at the contact positionwith the boundary plane, an arrangement may be made where a 3Dcoordinate region including the boundary plane is set, and difference indepth is calculated based on the z-axial coordinate values of pointsmaking up objects in the coordinate region. The largest value or averagevalue on the z-axis, and so forth, may be used for coordinates objectswithin the coordinate region as well.

The reference storage unit 175 stores the tolerance range for differencein depth that has been determined beforehand. The determination unit 176compares the difference in depth across the boundary plane that has beencalculated by the depth difference calculating unit 174 with thetolerance range stored in the reference storage unit 175, and determineswhether or not the difference in depth between images on the inside andthe outside of the boundary of the additional image display regionexceeds the tolerance range.

Description of Step S1200

FIGS. 10A through 10C illustrate examples of display region candidateswhich the display region candidate deciding unit 160 decides. FIG. 10Dis an example of an image acquired by the stereo endoscope camera 111,which is the main image. FIG. 10A illustrates an example of a candidateregion A, FIG. 10B illustrates an example of a candidate region B, andFIG. 10C illustrates an example of a candidate region C. The displayregion candidate deciding unit 160 decides the three candidate regions,candidate region A, candidate region B, and candidate region C. Thenumber of candidate regions may be any number of one or more. In stepS1200, the display region candidate deciding unit 160 decides displayregion candidates. Position information of the candidate regions isstored in the additional image position storage unit 140.

Operations of Depth Information Generating Unit 130 and Periphery DepthExtracting Unit 172: Details of Step S1300

In step S1300 the depth suitability determination unit 170 extracts aboundary line or boundary plane of a display region candidate of anadditional image decided in step S1200, and extracts depth information.The periphery depth extracting unit 172 extracts depth information ofthe main image at the periphery of the boundary plane from the depthinformation of the main image generated by the depth informationgenerating unit 130 and the boundary plane information obtained by theboundary extracting unit 171, in order to extract depth information ofthe boundary plane.

FIG. 11A is a schematic diagram illustrating boundary plane peripheryportion of the candidate region A illustrated in FIG. 10A. FIG. 11B is aleft-eye image of FIG. 11A, and FIG. 11C is a right-eye image of FIG.11A. FIG. 11B illustrates divided regions obtained by dividing theboundary region in FIG. 11A.

The depth information generating unit 130 extracts correlation points inthe left-eye image and the right-eye image. It is sufficient for theextraction of correlation points to be performed between steps S1100 andS1300. The correlation points are obtained by extracting the edges ofeach of the left-eye image and right-eye image, for example, and findingcorrelation from the positions and shapes of edges extracted in the leftand right images. Also, an arrangement may be made where color regionsof the left and right images are obtained, and the correlation pointsare found by finding correlation in the color regions.

A correlation point a and correlation point bin FIGS. 11B and 11C areexamples of correlation points in the left-eye image and right-eyeimage. The correlation point a is a similar positions on the screen inboth left and right images, while the position of the correlation pointb on the screen differs among the left and right images. The depthinformation generating unit 130 generates disparity at the left andright correlation points rendered on the screen, i.e., offsetinformation regarding the horizontal position in the left and rightimages. The periphery depth extracting unit 172 extracts correlationpoints in the boundary region in of one of the left and right images.The correlation points in the boundary region in the left-eye image areextracted in this example.

The periphery depth extracting unit 172 obtains the difference inhorizontal position between a certain point within the boundary regionof the left-eye image and a point in the right-eye image whichcorresponds to that point. The difference in horizontal position isrepresented as difference in X-coordinate on an x-y plane in acoordinate system such as illustrated in FIG. 2B, for example. A valueobtained by subtracting the value of the x-coordinate on the left-eyeimage from the value of the x-coordinate on the right-eye image is takenas the offset in horizontal position, i.e., the disparity. In this case,the point appears deeper than the plane of the screen of the disparityis positive, and the point appears nearer than the plane of the screenof the disparity is negative.

FIG. 12A is a schematic diagram illustrating a boundary region set on aleft-eye image. The boundary region is divided into partial regions, andan ID is set for each divided region. IDs of D1,1 through D7,10 areillustrated in the example in FIG. 12A.

FIG. 12B illustrates an example of disparity for each divided regionwhich the periphery depth extracting unit 172 has obtained from thecorrelation points extracted from within the boundary region in theleft-eye image. The periphery depth extracting unit 172 outputsinformation such as in FIG. 12B, for example, to the depth differencecalculating unit 174.

With regard to the depth of the main image and depth of the additionalimage at the boundary plane periphery of the additional image regionwhich the periphery depth extracting unit 172 has extracted, the depthdifference calculating unit 174 calculates the difference in depth ofthe main image and depth of the additional image for each boundaryplane. For example, for the boundary region of D1,1 through D1,10 inFIG. 12A, the depth is obtained from the largest disparity from D1,1through D1,10 which is a partial boundary region making up the x-z planeof the boundary between the additional image and main image. The depthdifference calculating unit 174 takes the difference between the depthof the boundary region which has been obtained and the depth of theadditional image as the difference in depth of the x-z boundary plane ofthe candidate region A of the additional image. The depth differencecalculating unit 174 performs similar calculations regarding the regionfrom D1,10 through D7,10 to calculate the difference in depth of the y-zboundary plane of the candidate region A. While description has beenmade here of calculating the depth of the boundary region from thegreatest disparity of the boundary region making up the boundary plane,the average disparity of the boundary region making up the boundaryplane, or the like, may be used instead.

Advantages and Effects

According to the second embodiment described above, at the time ofdisplaying an additional image over or adjacent to a 3D image from astereo endoscope, the 3D display device 20 decides the display region ofthe additional image avoiding states with great difference in depthbetween the inside and outside of the boundary of the display region ofthe additional image, and states with depth contradiction. In 3D imagesfor endoscope surgery using a stereo endoscope in particular, theaffected area to be treated by surgery is displayed around the middle ofthe screen, while the arms of forceps are often displayed at theperiphery portion of the screen. The forceps are often inserted towardthe affected area from the same direction as the stereo endoscope, andthus the arms thereof are imaged as slender objects extending in thedepth direction. Displaying the additional image of which the depth isclose to that of the plane of the display, so as to overlap the image ofthe forceps greatly protruding toward in the near side of the screen,results in an additional image which appears to have embedded itself inthe arms of the forceps. The above-described arrangement can avoid suchdepth contradiction and alleviate fatigue of the user, i.e., thesurgeon.

Description has been made regarding the second embodiment that thedisplay region candidate deciding unit 160 decides display regioncandidates not including the additional image display-forbidden regionstored in the display-forbidden region storage unit 210, and the depthsuitability determination unit 170 does not use information of thedisplay-forbidden region. However, an arrangement may be made where thedepth suitability determination unit 170 acquires distance indices ofthe distance from the display-forbidden region for each display regioncandidate, and gives display region candidates farther away from thedisplay-forbidden region priority as processing regions.

FIG. 13A is a schematic diagram illustrating a candidate region A, aboundary region, and an additional image display-forbidden region, onthe main image. FIG. 13B illustrates an example of information regardingdifference in depth for each candidate region which the display regioncandidate deciding unit 160 outputs to the depth suitabilitydetermination unit 170, and information of distance from the forbiddenregion. Information regarding the distance from the forbidden region,and information of difference in depth between the main image andadditional image in the periphery of the boundary, is included for eachcandidate region illustrated in FIGS. 10A through 10C.

In the example in FIG. 13B, first, the candidate region C is smallerthan −1 cm, which is the lower limit for the tolerance range indifference in depth, so this is not selected as a display region for theadditional image. Comparing the candidate region A and the candidateregion B shows that the difference in depth is smaller at the candidateregion B and thus the candidate region B is more suitable as a displayregion of the additional image, when the difference in depth alone istaken into consideration. However, the distance from the additionalimage display-forbidden region is smaller for the candidate region B ascompared to the candidate region A.

This can be dealt with by using corrected values, for example, where 5is added to the difference in depth for a range of up to 50 pixels fromthe additional image display-forbidden region, 3 is added for up to 75pixels, 2 is added to up to 100 pixels, and 1 is added to up to 200pixels. After 200 pixels, there is no correction by addition. Applyingsuch corrected values to the example in FIG. 13B gives 11 for thedifference in depth of the candidate region B since 5 has been added tothe difference in depth 6, and gives 9 for the difference in depth ofthe candidate region A since 2 has been added to the difference in depth7. In this case, the depth suitability determination unit 170 willselect the candidate region A as the optimal region to serve as thedisplay region for the additional image. Note that values for correctingdifference in depth based on distance from the additional imagedisplay-forbidden region, and the method of correction, are notrestricted to the above-described, and other methods may be used aswell.

Description has been made regarding the second embodiment that the depthsuitability determination unit 170 makes determination regarding whetheror not the difference between the depth of the main image and the depthof the additional image at the periphery of the boundary plane of acandidate region for an additional image display region is within thetolerance range. However, an arrangement may be made for cases where thedifference in depth only slightly exceeds the tolerance range, byproviding a frame wider than normal for the display region of theadditional image, so as to provide a buffer region regarding thedifference in depth, and determining the difference in depth with thebuffer region as a region where the additional image can be displayed.The term “slightly exceeds the tolerance range” means a case where, forexample, the amount of difference in depth exceeding the tolerancerange, i.e., the difference between the difference in depth and thetolerance range, is a predetermined value or smaller.

FIG. 14A illustrates an additional image in a case where the depthsuitability determination unit 170 has determined that the differencebetween the depth of the main image and the depth of the additionalimage at the periphery of the boundary plane of an additional imagedisplay region is within the tolerance range. The frame of thisadditional image is narrow. FIG. 14B illustrates an additional image ina case where the depth suitability determination unit 170 has determinedthat the difference between the depth of the main image and the depth ofthe additional image at the periphery of the boundary plane of anadditional image display region is not within the tolerance range, and abuffer region is necessary. The frame of this additional image is broad,serving as a buffer region regarding the difference in depth between theinside and outside of the frame.

One way to set a broader frame is to reduce the display size of theadditional image without changing the region in the main image which isshielded. If the region of the main image which is shielded can beenlarged, the frame can be made broader without changing the size of theadditional image. FIG. 14B illustrates a case where the additional imageis reduced in size and also the shielded region is enlarged.

First Modification of Second Embodiment

The endoscope camera 111 in the 3D display device 20 according to thesecond embodiment does not change the imaging position, and does notchange the distance between the endoscope camera 111 and the affectedarea, i.e., does not zoom in or zoom out. A case of changing thedistance between the endoscope camera 111 and the affected area will bedescribed in the present modification.

In a case of the endoscope camera 111 changing the distance as to theaffected area, the depth information of the 3D image changes. The regionis centered on the affected area where shielding by the additional imageis forbidden, also grows and shrinks accordingly. Accordingly, thedisplay region of the additional image also needs to change. Descriptionwill be made in the first modification of the second embodimentregarding a method of calculating the distance from the camera to theaffected area from images acquired by the endoscope camera 111, decodingthe additional image display-forbidden region according to the distance,and deciding the additional image display region.

FIG. 15 is a block diagram illustrating a functional configuration ofthe 3D display device 20 according to the first modification of thesecond embodiment. The configuration is the same as the 3D displaydevice 20 according to the second embodiment illustrated in FIG. 7,other than the display-forbidden region storage unit 210 has beenreplaced by a display-forbidden region storage unit 211, and a forbiddenregion deciding unit 212 has been added. Portions which are the same asthose in FIG. 7 are denoted by the same reference numerals, anddescription will be omitted.

Configuration

The 3D display device 20 includes the endoscope camera 111, vital signssensor 121, MRI image storage unit 122, 3D image compositing unit 123,depth information generating unit 130, additional image position storageunit 140, additional image size storage unit 150, display regioncandidate deciding unit 160, depth suitability determination unit 170,image compositing unit 180, display 190, display-forbidden regionstorage unit 211, forbidden region deciding unit 212, and input unit200.

The display-forbidden region storage unit 211 stores a region to forbiddisplay of an additional image. The forbidden region is around themiddle of the screen. The affected area which is to be treated bysurgery is shown around the middle of the screen, so in a case where theendoscope camera 111 draws closer to the affected area, a relativelywide range of the screen is an image of around the affected area, so thedisplay-forbidden region of the additional image becomes larger. On theother hand, in a case where the endoscope camera 111 is distanced fromthe affected area, only a relatively narrow range at the center portionof the screen is an image of around the affected area, so thedisplay-forbidden region of the additional image becomes smaller. Thedisplay-forbidden region storage unit 211 stores an additional imagedisplay-forbidden region corresponding to the distance between theendoscope camera 111 and the object at the center of the screen.

FIG. 16 illustrates an example of the contents stored in thedisplay-forbidden region storage unit 211. The display-forbidden regionstorage unit 211 stores ranges of distance between the endoscope camera111 and an object at the center of the screen, and additional imagedisplay-forbidden regions corresponding to the distance ranges. Theregions are represented as 3D image coordinates, such as in FIG. 2B. Thedisplay-forbidden region is represented as a region on an x-y plane, andthe value of the z axis is optional. A cylindrical region orthogonal tothe display plane is the display-forbidden region.

The forbidden region deciding unit 212 obtains the distance between theendoscope camera 111 and the object at the center of the screenfollowing the depth information of the main image output from the depthinformation generating unit 130. The forbidden region deciding unit 212extracts a display-forbidden region corresponding to the distancebetween the endoscope camera 111 and the object at the center of thescreen, from the information stored in the display-forbidden regionstorage unit 211, and decides the display-forbidden region of theadditional image. The display region of the additional image can thus bedecided, in accordance with change in the region where shielding by theadditional image is forbidden due to zoom in and zoom out operations ofthe endoscope camera 111.

The following is a detailed description of the depth informationgenerating unit 130 and of a method for calculating the distance to anobject at the center of the screen. FIG. 17 is a block diagramillustrating the details of part of the 3D display device 20 accordingto the first modification of the second embodiment.

The depth information generating unit 130 includes a correlation pointextracting unit 131, a disparity calculating unit 132, and a depthcalculating unit 133.

The correlation point extracting unit 131 obtains correlation points ina right-eye image left-eye image, from a stereo 3D image acquired at theendoscope camera 111. That is to say, points are extracted in theright-eye image and the left-eye image which have been taken of the samepoint in the same subject. Examples of correlation point extractioninclude a correlation method where correlation points are detected fromthe correlation in luminance distribution of the left and right images,and a method of obtaining correlation points by performing edgeextraction on the left and right images, for example.

The disparity calculating unit 132 extracts the position in theright-eye image and the position in the left-eye image for eachcorrelation point extracted by the correlation point extracting unit131, and obtains the difference in the horizontal direction between theleft and right images. For example, the correlation points in the leftand right images are illustrated following x-y plane coordinates wherethe horizontal direction on the display plane is the x axis, thevertical direction in the y axis, the right direction is the positive onthe x axis of coordinate axes having the center-of-gravity point of thedisplay as the origin, the left direction is negative on the x axis, theupward direction is positive on the y axis, and the downward directionis negative on the y axis. A value obtained by subtracting the xcoordinate value of the left-eye image from the x coordinate value ofthe right-eye image is the calculated disparity. If the object issituated on the display plane, the disparity is zero, and if the objectis situated nearer to the viewer from the display plane, the disparityis a negative value. If the object is situated on the far side of thedisplay plane from the viewer, the disparity is a positive value.

The depth calculating unit 133 calculates the distance between theendoscope camera 111 when imaging the object, and the depth position ofthe object in the image when displaying the object, based on thedisparity which the disparity calculating unit 132 has calculated. Thedepth calculating unit 133 outputs the distance between the endoscopecamera 111 and the object at the time of imaging to the forbidden regiondeciding unit 212, and outputs the depth position of the object in theimage at the time of displaying the image to the depth suitabilitydetermination unit 170.

While the distance between the endoscope camera 111 and the object atthe center of the screen, which is the affected area, has been describedin the first modification as being calculated using disparity of thestereo 3D image, an arrangement may be made where an object withunchanging size is determined in one or the other of the left and rightimages, and the distance between the endoscope camera 111 and the objectat the center of the screen is obtained based on the size of this objectin the image.

While the display-forbidden region storage unit 211 has been describedin the first modification as storing a display-forbidden regioncorresponding to the distance between the endoscope camera 111 and theobject, an arrangement may be made where the display-forbidden regionstorage unit 211 only stores a standard display-forbidden region, thesame as with the display-forbidden region storage unit 210. In thiscase, the forbidden region deciding unit 212 may enlarge or reduce thesize of the display-forbidden region stored in the display-forbiddenregion storage unit 211 based on the ratio between the distance from theendoscope camera 111 to the object and a standard distance.

Advantages

An additional image display-forbidden region corresponding to thedistance between the camera and affected area is stored in thedisplay-forbidden region storage unit 211, the distance between thecamera and affected area is calculated from an image acquired at theendoscope camera 111, and a display-forbidden region for the additionalimage is selected according to that distance. Accordingly, even if thedistance between the endoscope camera 111 and the affected area changesby the endoscope camera 111 zooming in or zooming out, and the region ofthe affected area and the periphery thereof in the image which shouldnot be shielded during surgery changes, the main image and additionalimage can be displayed without the additional image shielding theaffected area and the periphery.

Second Modification of Second Embodiment

While the second embodiment has been described as the depth of theadditional image being fixed, the depth of the additional image changesdepending on the content of display in the second modification. In acase where the depth of the additional image is fixed, in the secondembodiment the depth suitability determination unit 170 extracts onlythe depth range of the main image regarding the depth of the boundaryplane of the display region of the additional image and the peripherythereof, and compares with the depth of an additional image determinedbeforehand. However, in a case where the depth range of the additionalimage changes depending on the content of display as in the secondmodification, there is a need to calculate the depth range of theadditional image and perform comparison.

FIG. 18 is a block diagram illustrating a function configuration of the3D display device 20 according to the second modification of the secondembodiment. The configuration is the same as the 3D display device 20according to the second embodiment illustrated in FIG. 7, other than thepoints that the 3D image compositing unit 123 has been replaced by a 3Dimage compositing unit 124, the display region candidate deciding unit160 has been replaced by a display region candidate deciding unit 260,and the depth suitability determination unit 170 has been replaced by adepth suitability determination unit 270. Portions which are the same asthose in FIG. 7 are denoted by the same reference numerals, anddescription will be omitted.

Configuration

The 3D display device 20 includes the endoscope camera 111, vital signssensor 121, MRI image storage unit 122, 3D image compositing unit 124,depth information generating unit 130, additional image position storageunit 140, additional image size storage unit 150, display regioncandidate deciding unit 260, depth suitability determination unit 270,image compositing unit 180, display 190, display-forbidden regionstorage unit 210, and input unit 200.

The display region candidate deciding unit 260 selects an additionalimage size from additional image size information stored in theadditional image size storage unit 150, in accordance with additionalimage size specification information input at the input unit 200. Thedisplay region candidate deciding unit 260 also decides candidates foran additional image display region based on information of theadditional image position stored in the additional image positionstorage unit 140 and the information of the additional imagedisplay-forbidden region stored in the display-forbidden region storageunit 210, and outputs information indicating a decided candidate to thedepth suitability determination unit 270 and 3D image compositing unit124.

The 3D image compositing unit 124 composites the image informationstored in the MRI image storage unit 122 into an image which can bedisplayed in the additional image display region decided at the displayregion candidate deciding unit 260, based on instructions regardingdisplay contents input from the input unit 200, such as for exampledisplay range, display scale, type of MRI image (whether 2D image or 3Dimage), and so forth. The 3D image compositing unit 124 further outputsdepth information of the composited image to the depth suitabilitydetermination unit 270.

The depth suitability determination unit 270 extracts the boundary planeof the additional image display region based on information of thecandidate of the display region decided by the display region candidatedeciding unit 260, and extracts the depth of the main image at theperiphery of the boundary plane from the depth information of the mainimage generated at the depth information generating unit 130. The depthsuitability determination unit 270 further extracts depth information ofthe additional image at the periphery of the boundary plane from thedepth information of the image which the 3D image compositing unit 124has composited. The depth suitability determination unit 270 comparesthe difference in depth between the main image side and additional imageside of the boundary plane of the additional image display region thathas been extracted, and detects difference in depth exceeding apredetermined value or depth contradiction.

FIG. 19 is a block diagram illustrating part of the 3D display device 20according to the second modification of the second embodiment in detail.The configuration in FIG. 19 that in FIG. 9 according to the secondembodiment, other than the points that the periphery depth extractingunit 172 has been replaced by a periphery depth extracting unit 272, andthe additional image storage unit 173 has been omitted. Portions whichare the same as those in FIG. 9 are denoted by the same referencenumerals, and description will be omitted.

The depth suitability determination unit 270 includes the boundaryextracting unit 171, periphery depth extracting unit 272, depthdifference calculating unit 174, reference storage unit 175, anddetermination unit 176. The periphery depth extracting unit 272 extractsdepth information of the main image at the boundary plane periphery fromthe depth information of the main image which the depth informationgenerating unit 130 has generated, and the boundary plane informationwhich the boundary extracting unit 171 has obtained. The periphery depthextracting unit 272 further extracts the depth information of theadditional image at the boundary plane periphery from the depthinformation of the additional image generated by the 3D imagecompositing unit 124 and the boundary plane information obtained by theboundary extracting unit 171.

FIG. 20A illustrates an additional image display candidate region, aboundary region on a main image which is a peripheral region of anadditional image display region, and a boundary plane adjacent regionwhich is an additional image side boundary plane peripheral region. FIG.20B illustrates an example of each divided region and IDs thereofregarding the boundary region and boundary plane adjacent region in FIG.20A.

The method for obtaining the depth of the boundary region is the same aswith the second embodiment. A method for calculating depth informationof the boundary plane adjacent region will now be described. The 3Dimage compositing unit 124 acquires information of the display regionfor the additional image display region candidate from the displayregion candidate deciding unit 260, and generates image information fordisplaying a computer graphics 3D image at the specified display region.The image information includes the correlation points for the left-eyeimage and right-eye image, and disparity information for the correlationpoints.

The periphery depth extracting unit 272 extracts a boundary within apredetermined distance from the boundary plane as to the main image asthe boundary plane adjacent region, for one or the other of the left-eyeimage and right-eye image, as illustrated in FIG. 20A. The peripherydepth extracting unit 272 further divides the boundary plane adjacentregion into partial regions as illustrated in FIG. 20B, for example. Inthis case, the boundary plane adjacent region has been set on theleft-eye image, and partial region (boundary region) C1 through partialregion (boundary region) C3 have been set. Note that a boundary planeadjacent region and partial regions thereof may be set on the right-eyeimage. The periphery depth extracting unit 272 extracts the disparitybetween a point making up an object displayed within the boundary planeadjacent region and a correlation point on the right-eye image, from theimage information which the 3D image compositing unit 124 has generated.The periphery depth extracting unit 272 obtains the greatest disparityvalue in disparity at each point of the object, for each partial region.While the greatest value is used here as a representative value ofdisparity in the partial regions, other statistical values may be used,such as average value or the like.

Note that the main image has a broad range of depth, and change in depthis marked. Accordingly, there are cases where difference in depth may begreat between adjacent boundary regions. Reducing the size of theboundary regions enables the depth of the boundary regions to becalculated accurately. On the other hand, the additional image has anarrow range of depth, and change is gradual. Accordingly, the depth atthe boundary plane adjacent region can be accurately calculated even ifthe size of the boundary plane adjacent regions is relatively large.

FIG. 21 is an example of information output from the periphery depthextracting unit 272. Along with the boundary region and boundary planeadjacent region, the largest value of disparity for each partial regionis extracted as a representative value of disparity of the region. Basedon disparity information such as illustrated in FIG. 21, the depthdifference calculating unit 174 calculates the depth position of eachregion from the disparity of which the absolute value is the largestfrom boundary region D1,1 to D1,4, and the disparity of boundary planeadjacent region C1, and obtains the difference in depth between theboundary region which is at the main image side and the boundary planeadjacent region which is at the additional image side. In the same way,the depth difference calculating unit 174 obtains the difference indepth for each of boundary region D1,5 to D1,10, and boundary planeadjacent region C2, and boundary region D2,10 to D7,10, and boundaryplane adjacent region C3.

FIG. 22 is a block diagram illustrating part of the 3D display device 20according to the second modification of the second embodiment in detail.The display region candidate deciding unit 260 has an additional imageposition selecting unit 261, an additional image size selection unit 262and a region deciding unit 263.

The additional image position selecting unit 261 selects information ofadditional image positions of a number specified by the input unit 200,from additional image position information stored in the additionalimage position storage unit 140.

The additional image size selecting unit 262 selects size information ofthe additional image from information stored in the additional imagesize storage unit 150, following the additional image size informationspecified at the input unit 200.

The region deciding unit 263 decides a candidate for a region to displaythe additional image, based on the additional image position selected bythe additional image position selecting unit 261 and the additionalimage size selected at the additional image size selecting unit 262, andoutputs this to the 3D image compositing unit 124 and the depthsuitability determination unit 270.

Advantages

The display region candidate deciding unit 260 decides a candidate foran additional image region following additional image size information,out of display conditions for the additional image input at the inputunit 200. On the other hand, the 3D image compositing unit 124composites the additional image at the image size which the displayregion candidate deciding unit 260 has decided, following the displayconditions for the additional image input at the input unit 200. Thedepth suitability determination unit 270 extracts main image side depthinformation for the boundary plane of the additional image displayregion from the depth information generating unit 130, and additionalimage side depth information from the 3D image compositing unit 124, anddecides whether or not the difference in depth across the boundary planeof the display region is within a suitable range. Due to thisconfiguration, even if the depth of an image displayed as an additionalimage changes due to user instructions, the display range of theadditional image can be set so that the difference in depth across theboundary of the display region of the additional image does not exceedthe suitable range.

Third Modification of Second Embodiment

While the display-forbidden region storage unit 213 according to thesecond embodiment only stores a predetermined forbidden region, thedisplay-forbidden region storage unit 213 according to a thirdmodification sets and stores a new display-forbidden region from animage imaged by the endoscope camera 111. In a case of endoscopicsurgery, the endoscope camera and surgical tools are inserted into thebody through multiple guide tubes. While the position and direction ofthe guide tubes differ from one surgery to another, the guide tubeshardly every move during surgery. Accordingly, while there may bedifferences such as whether a surgical instrument is inserted or notinserted during the surgery, but the positional relation between thecamera and inserted surgical instrument, and the positional relationbetween the camera and the structure within the body does not changegreatly. Accordingly, images are detected which have a depth such thatthere is a possibility that depth contradiction or excessively largedepth difference will occur between the main image imaged by theendoscope camera 111 after surgery starts and the additional image whichshields a part of the main image, and such detected images are stored asadditional image display-forbidden regions. Thus, the search range forthe additional image display region can be restricted, thereby reducingthe load of the additional image display region deciding. Further,movement of the additional image display region can be reduced, so thedisplay of the additional image stabilizes.

FIG. 23 is a block diagram illustrating a part of the 3D display device20 according to the third modification of the second embodiment indetail. The configuration in FIG. 23 is the same as the firstmodification of the second embodiment in FIG. 17, other than the pointsthat the display-forbidden region storage unit 211 has been replaced bya display-forbidden region storage unit 213, and the forbidden regiondeciding unit 212 has been replaced by a forbidden region deciding unit214. Portions which are the same as those in FIG. 17 are denoted by thesame reference numerals, and description will be omitted.

The 3D display device 20 according to the third modification of thesecond embodiment is the same as that according to the first embodimentof the second embodiment in FIG. 15, other than the points that thedisplay-forbidden region storage unit 211 has been replaced by adisplay-forbidden region storage unit 213, and the forbidden regiondeciding unit 212 has been replaced by a forbidden region deciding unit214. Accordingly, description of the block diagram illustrating thefunctions of the 3D display device 20 will be omitted.

The display-forbidden region storage unit 213 stores information ofregions to forbid display of additional images. That is to say, thedisplay-forbidden region storage unit 213 stores, as information ofregions to forbid display of additional images, the information of theregion around the center of the screen that has been determinedbeforehand, and information of portions where depth is markedly great,i.e., portions deeply recessed, and where depth is markedly small, i.e.,portions greatly protruding, detected from the image imaged by theendoscope camera 111.

The forbidden region deciding unit 214 extracts information of regionsto forbid display of the additional images, stored in thedisplay-forbidden region storage unit 213, and outputs to the displayregion candidate deciding unit 160. The forbidden region deciding unit214 further decides a new forbidden region, based on information of adisplay region including depth information of an object or backgroundimaged by the endoscope camera 111, that has been generated at the depthinformation generating unit 130, and outputs the decided forbiddenregion to the display-forbidden region storage unit 213.

Deciding of the new forbidden region is performed by the forbiddenregion deciding unit 214 as follows. In a case where a region which isrecessed to the deep side beyond the predetermined depth range, or aregion protruding to the near side beyond the predetermined depth range,are not included in the display-forbidden region stored in thedisplay-forbidden region storage unit 213, this region is a newdisplay-forbidden region. For example, in a case of imaging a tubularstructure such as a digestive organ in the longitudinal direction, themain image will be an image looking through the tubular structure, sothe recessed amount to the deep side will be extremely great. If thedepth exceeds the maximum depth of the depth range for the additionalimage and is larger than a predetermined value, there is a possibilitythat the difference in depth between the additional image and the mainimage may be too great. Such a region is made to be an additional imagedisplay-forbidden region.

For example, in a case where the depth range of the additional image is−5 cm on the z axis in FIG. 2B, which is the far side of the display, to5 cm on the z axis which is the near side of the display, an object orbackground at −20 cm or deeper on the z axis from the display may havedifference in depth between the main image and additional image thatexceeds 15 cm. Accordingly, that region is set to an additional imagedisplay-forbidden region. Also, an object such as a forceps arm forexample, at 16 cm or nearer on the z axis from the display, may havedepth contradiction between the main image and additional image thatexceeds −1 cm. Accordingly, that region is set to an additional imagedisplay-forbidden region.

Advantages

Portions having depth contradiction and excessively large difference indepth between the main image and additional image, due to bodystructures and inserting of medical instruments such as forceps and thelike are handled as follows. Regions where determination is made thatthe detected depth position of body structures and medical instrumentsimaged in the main image may exceed the depth suitability range arestored as additional image display-forbidden regions, where noadditional images are displayed even in a state with the medicalinstruments and the like removed. Accordingly, this does away withtrouble such as a case where removal of inserted forceps temporarilyresolves depth contradiction and so forth between the main image andadditional image, the additional image is displayed on the region wherethe depth contradiction and so forth has been temporarily resolved, theforceps are inserted again resulting in depth contradiction, so thedisplay position of the additional image is greatly changed. The searchregion range for the additional image display region also becomesnarrower, so the load due to calculation is reduced.

Fourth Modification of Second Embodiment

While an arrangement has been described in the third modification of thesecond embodiment where the additional image display-forbidden regiondetermined beforehand and the additional image display-forbidden regiondetermined from an image imaged by the endoscope camera 111 are handledin the same way, the additional image display-forbidden regiondetermined beforehand and the additional image display-forbidden regiondetermined from an imaged image may be handled separately. Theadditional image display-forbidden region determined beforehand is theaffected area to be treated by surgery and the perimeter thereof, at thecenter of the screen, and is a region where the main image must bedisplayed during surgery without fail. However, the additional imagedisplay-forbidden region determined from an imaged image is a regionwhere depth contradiction may occur or difference in depth may be toogreat between the main image and the additional image. Since theadditional image display-forbidden region determined from an imagedimage is a region there depth contradiction may occur or difference indepth may be too great, so shielding such a region by an additionalimage can prevent depth contradiction or excessively great difference indepth from occurring. Note that in the present modification, a regionwhere depth contradiction may occur or difference in depth may be toogreat will be referred to as a “possibly-unsuitable region”.

Configuration

FIG. 24 is a block diagram illustrating a part of a 3D display deviceaccording to a fourth modification of the second embodiment in detail.The configuration is the same as that in FIG. 22, other than the pointsthat the display-forbidden region storage unit 213 has been replaced bya display-forbidden region storage unit 215, the display regioncandidate deciding unit 160 has been replaced by a display regioncandidate deciding unit 161, the forbidden region deciding unit 214 hasbeen replaced by a possibly-unsuitable region deciding unit 217, and apossibly-unsuitable region storage unit 216 has been added. Portionswhich are the same as those in FIG. 22 are denoted by the same referencenumerals, and description will be omitted.

The display-forbidden region storage unit 215 stores information of theadditional image display-forbidden region determined beforehand.

The possibly-unsuitable region deciding unit 217 decides, from the depthinformation of an image imaged by the endoscope camera 111 that has beenoutput from the depth information generating unit 130, a region wherethe possibility is great that depth contradiction will occur between themain image and additional image, or where the possibility is great thatthe difference in depth will be too great. That is to say, thepossibly-unsuitable region deciding unit 217 decides regions in the mainimage where the depth protrudes too far in the near side beyond apredetermined depth range, and regions where the depth is recessed toofar in the deep side beyond the predetermined depth range.

The possibly-unsuitable region storage unit 216 stores regions decidedby the possibly-unsuitable region deciding unit 217.

The display region candidate deciding unit 161 decides a display regionfor the additional image, based on position information of theadditional image which the additional image position storage unit 140has acquired, size information of the additional image which theadditional image size storage unit 150 has acquired, the additionalimage display-forbidden region that has been determined beforehand,acquired by the display-forbidden region storage unit 215, andinformation of regions in the image imaged by the endoscope camera 111where the possibility is high that depth contradiction will occur or thedifference in depth will be too great, acquired from thepossibly-unsuitable region storage unit 216.

Operations

FIG. 25 is a flowchart illustrating operations of the display regioncandidate deciding unit 161 deciding a display region for an additionalimage. The operations of FIG. 25 are a detailed illustration of theoperations equivalent to steps S1200 and S1210 in FIG. 8, in a casewhere the additional image display-forbidden region determinedbeforehand and the additional image display-forbidden region determinedfrom an imaged image are handled separately.

The display region candidate deciding unit 161 first sets the additionalimage size according to the size information of the additional imagestored in the additional image size storage unit 150 (step S1201).

The display region candidate deciding unit 161 next compares thepossibly-unsuitable region acquired from the possibly-unsuitable regionstorage unit 216 with the size of the additional image set in stepS1201, and determines whether the possibly-unsuitable region can beshielded by the additional image (step S1202). In a case wheredetermination is made in step S1202 that the possibly-unsuitable regioncan be shielded by the additional image, i.e., step S1202 yields “yes”,the flow advances to step S1203. In a case where determination is madein step S1202 that the possibly-unsuitable region cannot be shielded bythe additional image, i.e., step S1202 yields “no”, the flow advances tostep S1204.

The display region candidate deciding unit 161 sets the position of theadditional image based on the position information stored in theadditional image position storage unit 140 (step S1203).

The display region candidate deciding unit 161 then determines whetheror not the region of the additional image, decided by the size of theadditional image set in step S1201 and position of the additional imageset in step S1203, includes a region regarding which display of anadditional image has been forbidden, stored in the display-forbiddenregion storage unit 215 (step S1210 a).

In a case where the set additional image region is determined in stepS1210 a to include a display-forbidden region, i.e., step S1210 a yields“yes”, the flow returns to step S1203. In a case where the setadditional image region is determined in step S1210 a to not include adisplay-forbidden region, i.e., step S1210 a yields “no”, the flowadvances to step S1208.

On the other hand, the display region candidate deciding unit 161determines whether or not the size of the additional image set in stepS1201 is the limit value of the additional image size set beforehand instep S1201 or larger (step S1204). In a case where determination is madein step S1204 that the size of the additional image is equal or largerthan the limit value, i.e., step S1204 yields “yes”, the flow advancesto step S1205. In a case where determination is made in step S1204 thatthe size of the additional image smaller than the limit value, i.e.,step S1204 yields “no”, the flow returns to step S1201.

In step S1205, the display region candidate deciding unit 161 referencesthe additional image size storage unit 150 and resets the size of theadditional image (step S1205).

The display region candidate deciding unit 161 further sets the positionof the additional image based on the position information stored in theadditional image position storage unit 140 (step S1206).

The display region candidate deciding unit 161 then determines whetheror not the region of the additional image decided by the size of theadditional image reset in step S1205 and the position of the additionalimage set in step S1206 includes a region regarding which display of anadditional image has been forbidden, stored in the display-forbiddenregion storage unit 215 (step S1210 b).

In a case where the set additional image region is determined in stepS1210 b to include a display-forbidden region, i.e., step S1210 b yields“yes”, the flow returns to step S1206. In a case where the setadditional image region is determined in step S1210 b to not include adisplay-forbidden region, i.e., step S1210 b yields “no”, the flowadvances to step S1207.

The display region candidate deciding unit 161 then determines whetheror not the set region of the additional image includes a region storedin the possibly-unsuitable region storage unit 216 (step S1207).

In a case where the additional image display region is determined instep S1207 to include a possibly-unsuitable region, i.e., step S1207yields “yes”, the flow returns to step S1206. In a case where theadditional image display region is determined in step S1207 to notinclude a possibly-unsuitable region, i.e., step S1207 yields “no”, theflow advances to step S1208.

In step S1208, the display region candidate deciding unit 161 decidesthe additional image display region of the set size and position to be adisplay region candidate (step S1208).

Advantages and Effects

As described above, a region in the main image where there is apossibility of depth contradiction or excessively large difference indepth occurring between the main image and additional image is shieldedby the additional image, by adjusting the size and position of theadditional image. In a case where this region in the main image cannotbe shielded, the additional image is displayed at a position avoidingthe region where there is a possibility of depth contradiction orexcessively large difference in depth occurring between the main imageand additional image. Thus, a natural 3D image which does not place aload on the user can be presented.

While description has been made in the fourth modification of the secondembodiment that the size of the additional image is automaticallyadjusted, the size of the additional image may be adjusted only in acase where there is input from the input unit 200 to change the size ofthe additional image. Particularly, the processing to shield a region inthe main image where there is a possibility of depth contradiction orexcessively large difference in depth occurring between the main imageand additional image by the additional image may be performed in a casewhere there is input instructing enlarging the size of the additionalimage.

Change of the size of the additional image may be made such that aportion of the periphery of the screen, where the arm portions offorceps that have been imaged are an extreme protruding image, isshielded by extending the shape of the additional image in thehorizontal direction. Thus, regions on the right-eye image and left-eyeimage that correspond to an object with large disparity due to extremeprotrusion can be shielded in both images.

Fifth Modification of Second Embodiment

Description has been made in the second embodiment that the 3D displaydevice 20 avoids the display-forbidden region stored in thedisplay-forbidden region storage unit 210 based on the display positionof the additional image stored in the additional image position storageunit 140, without the user operating the position of the additionalimage, and the additional image is moved to a position where unsuitabledepth at the periphery of the boundary plane of the additional imagedisplay region can be avoided. A case will be described in the presentmodification where the user specifies the size of the additional imageby input at the input unit 200.

The configuration of the 3D display device 20 according to a fifthmodification is the same as that of the second embodiment, except thatthe output of the input unit 200 is input to the additional image sizestorage unit 150. Accordingly, illustration by way of drawings anddescription thereof will be omitted.

FIG. 26 is a flowchart illustrating the operations of the 3D displaydevice 20 according to the fifth modification of the second embodiment.FIG. 26 is the same as FIG. 8 in the second embodiment, other than stepsS1180 and S1190 having been added. Portions which are the same as thosein FIG. 8 will be denoted with the same reference numerals, anddescription will be omitted.

First, the endoscope camera 111 generates image information for 3Ddisplay having left and right disparity, as a main image, and the vitalsigns sensor 121 measures the current cardioelectric potential and bloodpressure of the patient as additional information (step S1100).

The input unit 200 acquires operations according to user input, anddetects instruction input to display an additional image (step S1110).In a case where additional image display instruction input is detectedin step S1110, i.e., in a case where step S1110 yields “yes”, the flowadvances to step S1180. In a case where additional image displayinstruction input is not detected in step S1110, i.e., in a case wherestep S1110 yields “no”, the flow advances to step S1700.

In step S1700, the image compositing unit 180 composites an endoscopecamera image which is a main image with no additional image as a displayimage.

In step S1180, the input unit 200 further detects input of operating thesize of the additional image (step S1180). In a case where inputspecifying the size of the additional image is detected in step S1180,i.e., in the case step S1180 yields “yes”, the flow advances to stepS1190. In a case where input specifying the size of the additional imageis not detected in step S1180, i.e., in the case step S1180 yields “no”,the flow advances to step S1200.

In step S1190, the input unit 200 outputs the information of size of theadditional image in the instruction input relating to display of theadditional image acquired in step S1110 to the additional image sizestorage unit 150. The additional image size storage unit 150 stores theinformation of the size of the additional image which the input unit 200has output (step S1190).

The display region candidate deciding unit 160 decides one candidate forthe display region, from the size of the additional image stored in theadditional image size storage unit 150, and the position of theadditional image stored in the additional image position storage unit140 (step S1200). For example, the display region candidate decidingunit 160 arbitrarily selects one unselected combination of additionalimage size stored in the additional image size storage unit 150 andadditional image position stored in the additional image positionstorage unit 140, and decides a region represented by the selectedcombination as a display region candidate.

Note that the newest information of the additional image informationstored in the additional image size storage unit 150 is used in stepS1200. Accordingly, in a case where step S1190 has been executed, thesize information recorded in step S1190 is used to decide the displayregion candidate.

The display region candidate deciding unit 160 determines whether or notthe display region candidate decided in step S1200 includes thedisplay-forbidden region stored in the display-forbidden region storageunit 210 (step S1210). In a case where determination is made in stepS1210 that the display region candidate includes the display-forbiddenregion, i.e., in a case where step S1210 yields “yes”, the flow advancesto step S1600.

In step S1600, the display region candidate deciding unit 160 moves thedisplay position of the additional image and stores the moved displayposition in the additional image position storage unit 140.

In a case where determination is made in step S1210 that the displayregion candidate does not include the display-forbidden region, i.e., ina case where step S1210 yields “no”, the flow advances to step S1300.

Repeating steps S1200, S1210, and S1600 decides the display regioncandidate while avoiding the display-forbidden region. The depthsuitability determination unit 170 extracts a boundary line or boundaryplane of the additional image display region candidate decided in stepS1200, and extracts depth information in the periphery of the boundaryline or boundary plane (step S1300).

The depth suitability determination unit 170 further compares the depthof the display region of the additional image obtained from the positioninformation of the additional image stored in the additional imageposition storage unit 140 with the depth of the main image at theportion adjacent to the boundary line or boundary plane of theadditional image extracted in step S1300 (step S1400).

The depth suitability determination unit 170 determines whether or notthe difference in depth of the main image and additional image displayedacross the boundary line or boundary plane exceed the tolerance range(step S1500). In a case where determination is made in step S1500 thatthe difference in depth exceeds the tolerance range, i.e., in a casewhere step S1500 yields “yes”, the flow advances to step S1600, andafter executing step S1600, returns to step S1200. In a case wheredetermination is made in step S1500 that the difference in depth iswithin the tolerance range, i.e., in a case where step S1500 yields“no”, the flow advances to step S1700.

In step S1700, the image compositing unit 180 composites the main imageacquired in step S1100 and the additional image representing theadditional information acquired in step S1100. Specifically, the imagecompositing unit 180 displays the main image on the screen of thedisplay 190, and displays the additional image so as to be displayed inthe display region which is the display region candidate decided in stepS1200, thus compositing the image.

The display 190 displays the 3D image composited in step S1700 (stepS1800), and subsequently the flow returns to step S1100. Repeating stepsS1100 through S1800 causes the 3D display device 20 to main acquireimages and additional information in increments of processing, andcontinue displaying of images. The present embodiment is applicable incases where the main image and additional image are moving images, aswell.

Now, the input unit 200 may be a switchbox such as illustrated in FIG.27A, for example. In this arrangement, the user can select the size bypressing one of multiple buttons in the switchbox corresponding tomultiple sizes (large, medium, and small buttons). Alternatively, theinput unit 200 may be a pointing device such as a mouse or a stylus. Inthe example of an additional image size specifying screen illustrated inFIG. 27B, the user can move an arrow shown at a vertex of the additionalimage region by operating a pointing device. This, the additional imageon the screen can be enlarged or reduced at will, and an operationalsize can be specified.

In another arrangement, the input unit 200 may include a camera whichshoots the user, and an image processing device which distinguishes useroperations from images of the user shot by the camera. This enables useroperations to specify size. In another arrangement, the input unit 200may include an input device whereby the user can input planar or 3Dcoordinate positions, direction of movement, and amount of movement,such as a touch panel or the like. In a case where the additional imageis rectangular, the size may be specified by inputting the height andwidth of the sides as numerical values to specify the size. A text inputdevice capable of inputting numerical values, such as a keyboard, can beused as the input unit 200 of the 3D display device 20. The input unit200 of the 3D display device 20 further may be a touch panel, ahandwriting recognition device which can take stylus input, or an audioinput device capable of audio input. The size of the additional imageregion can be specified for shapes other than rectangles, by specifyingthe length of sides, diagonal lines, or the like.

Advantages

According to the fifth modification of the second embodiment describedabove, at the time of displaying an additional image over or adjacent toa 3D image from a stereo endoscope, the 3D display device 20 decides thedisplay region of the additional image upon the user specifying thedisplay size of the additional image, avoiding states with greatdifference in depth between the inside and outside of the boundary ofthe display region of the additional image, and states with depthcontradiction. This enables the additional image to be displayed at asize which the user, i.e., the surgeon needs, while avoiding depthcontradictions such as an additional image which appears to haveembedded itself in the arms of the forceps, thereby alleviating fatigueof the user, i.e., the surgeon.

Sixth Modification of Second Embodiment

Description has been made in the fifth modification of the secondembodiment regarding the 3D display device 20 which decides sizesfollowing size instruction for additional images input from the inputunit 200. The input unit 200 according to the fifth modification of thesecond embodiment includes an input device for input of planar or 3Dcoordinate positions, direction of movement, and amount of movement, andincludes an input interface device used in a general environment.Description will be made in the sixth modification of the secondembodiment regarding a method of instructing change of the size ofadditional image, based on motion of forceps.

FIGS. 28A and 28B are diagrams schematically illustrating an arrangementwhere forceps are incorporated as a part of the input unit 200, and thesize of the additional image display region is adjusted based on themovement of the forceps. FIGS. 28A and 28B show a main image of a wirelooped over the affected area along with the forceps, and vital signsdata is illustrated in a graph in which the horizontal axis is passageof time. The vital signs data is the additional image displayedsuperimposed on the main image. None of the forceps tips shown at theleft side in FIGS. 28A and 28B overlap the additional image. In FIG.28B, the region of the additional image is smaller as compared to FIG.28A. When the surgeon pulls the forceps to the near side for example,the tips pass through the display region of the additional image (theboundary between the additional image and the main image). In a casewhere the forceps tips passage from outside of the display region of theadditional image to inside the display region of the additional image isdetected, the input unit 200 changes the shape of the additional imagein accordance with the direction of movement of the forceps. In a casewhere the forceps tips move in a direction from the outside of theregion toward the inside of the region, the movement of the tips isregarded as being input to reduce the size of the additional imagedisplay region, and in a case where the forceps tips move in a directionfrom the inside of the region toward the outside of the region, themovement of the tips is regarded as being input to enlarge the size ofthe additional image display region.

FIG. 29 is a block diagram illustrating a functional configuration ofthe 3D display device 20 according to the sixth modification of thesecond embodiment. The configuration in FIG. 29 is the same as that ofthe 3D display device 10 according to the first embodiment illustratedin FIG. 2, except for addition of a display region storage unit 401, aninstrument tip detecting unit 402, a passage detecting unit 403, anoperating input unit 404, and a display control signal generating unit405, as functional components of an input unit 400. Portions which arethe same as those in FIG. 7 are denoted with the same reference numeralsas in FIG. 7, and description thereof will be omitted.

The 3D display device 20 includes the endoscope camera 111, vital signssensor 121, MRI image storage unit 122, 3D image compositing unit 123,depth information generating unit 130, additional image position storageunit 140, additional image size storage unit 150, display regioncandidate deciding unit 160, depth suitability determination unit 170,image compositing unit 180, display 190, display-forbidden regionstorage unit 210, and input unit 400. The input unit 400 includes thedisplay region storage unit 401, instrument tip detecting unit 402,passage detecting unit 403, operating input unit 404, and a displaycontrol signal generating unit 405.

The endoscope camera 111 is a 3D imaging endoscope camera including astereo camera.

The vital signs sensor 121 is a sensor attached to the body of thepatient during surgery. In the sixth modification of the secondembodiment, description will be made with the vital signs sensor 121serving as an electrocardiogram and a sphygmomanometer.

The MRI image storage unit 122 stores 3D image information includingimages of the affected area to be treated by surgery, which have beenrecorded by an MRI system before surgery.

The 3D image compositing unit 123 composites image information stored inthe MRI image storage unit 122 into images of a format which can bedisplayed on the display 190, as specified slices or a spectrogram of aspecified range.

The additional image position storage unit 140 stores the position wherethe additional information is to be displayed on the screen as anadditional image.

The additional image size storage unit 150 stores the size at which theadditional information is to be displayed on the screen as an additionalimage.

The display-forbidden region storage unit 210 stores informationrepresenting a region determined beforehand where an affected area to besubjected to surgery has been photographed, as an additional imagedisplay-forbidden region. The display-forbidden region in the sixthmodification of the second embodiment is a fixed rectangular regioncentered on the center of the screen.

The display region candidate deciding unit 160 decides display regioncandidates to display one or more of additional information on thescreen of the display 190 as additional images. The additionalinformation in the present modification is the two types of vital signsinformation acquired by the vital signs sensor 121 during surgery, andMRI image information recorded before the surgery. These two types ofadditional information are each to be displayed at display regions, asseparate additional images.

The depth suitability determination unit 170 detects difference in depthgreater than a predetermined value or depth contradiction, with regardto difference in depth between the display regions of the additionalimages and the portions of the main image which are at the periphery ofthe display regions of the additional images, based on information ofdisplay region candidates which the display region candidate decidingunit 160 has decided, and depth information of the main image which thedepth information generating unit 130 has generated. The tolerance rangeis, for example −1 cm to 15 cm.

The image compositing unit 180 composites the 3D image and additionalimages such that the 3D image acquired by the endoscope camera 111 isdisplayed on the screen of the display 190, and the additionalinformation acquired by the vital signs sensor 121 is displayed asadditional images in at least one of the regions decided by the displayregion candidate deciding unit 160.

The display 190 displays the image composited by the image compositingunit 180.

The operating input unit 404 is a unit used by the user to instructwhether or not to display an additional image, and to input conditionsfor display of additional images.

The display control signal generating unit 405 acquires operations madeby the user regarding the additional image input from the operatinginput unit 404, and generates control signals for the image compositingunit 180 to control image compositing.

The display region storage unit 401 stores a display region whereadditional information is currently displayed as an additional image.FIG. 30 illustrates an example of contents stored in the display regionstorage unit 401. The display region storage unit 401 stores IDs ofadditional image regions, the contents of the additional information foreach ID, and three-dimensional coordinate positions of the four verticesfor describing the display region as a rectangular plane.

The instrument tip detecting unit 402 detects the tips of surgicalinstruments in the image imaged by the endoscope camera 111. An examplewill be given here where forceps tips are detected. The method ofdetection is as follows. The instrument tip detecting unit 402 extractsmetallic-colored regions as instrument regions from the color regions ofthe image imaged by the endoscope camera 111. The instrument tipdetecting unit 402 obtains correlation points in the left and rightimages regarding the extracted regions. The instrument tip detectingunit 402 obtains the depth of the instrument regions from thecorrelation points that have been obtained. The instrument tip detectingunit 402 extracts the deepest part of a color region where the color iscontinuous as being the tip portion. Alternatively, after extracting ametallic-colored region, the outline of the extracted region and theoutline of an instrument prepared beforehand may be subjected to patternmatching, thus deciding the tip. Other types of image processing may beperformed to detect the tip of the instruments, as well.

The passage detecting unit 403 detects that the tip portion of theinstrument detected by the instrument tip detecting unit 402 has passedthrough the additional image display region indicated by informationstored in the display region storage unit 401. The method of detectionis as follows. The passage detecting unit 403 extracts athree-dimensional coordinate position of the tip of the instrument whichthe instrument tip detecting unit 402 has detected, based onthree-dimensional coordinate position of an object in the imagegenerated by the depth information generating unit 130. For example, ina case of detecting that the three-dimensional coordinates of the tip ofthe instrument have passed through or come into contact with anadditional image plane stored in the display region storage unit 401,the passage detecting unit 403 detects that the instrument tip haspassed through the additional image region. When performing thisdetection, the passage detecting unit 403 also determines the directionof passage of the instrument tip, which is to say whether the instrumenttip is moving in a direction from inside the additional image regiontoward the outside of the region, or moving in a direction from outsidethe additional image region toward the inside of the region. The passagedetecting unit 403 outputs the information of the instrument tip passingthrough the additional image plane to the display control signalgenerating unit 220 along with the direction of passage.

Description has been made there that the passage detecting unit 403 usesthree-dimensional coordinates to detect passage of the instrument tipthrough the additional image region, but an arrangement may be madewhere the three-dimensional coordinates of the additional image regionand the three-dimensional coordinates of the instrument tip are mappedto the display plane, and passage of the instrument tip on thetwo-dimensional coordinates of the display plane is detected. In a casewhere the tip position of an instrument mapped to the display plane iswithin or in contact with the additional image region mapped to thedisplay play, the instrument tip may be regarded to have passed throughthe additional image, and detection of passage thus performed.

The passage of the instrument tip through the additional image regionand the passage direction thereof identifies an additional imageregarding which the size is to be specified, and also indicates sizechanging information of this additional image. When changing the size,the amount of size change or the size change ratio per change may bemade constant. Alternately, the amount of size change may be decided bythe moving speed and moving distance of the instrument tip. The passagedetecting unit 403 outputs to the display control signal generating unit405 information of size specification input for a particular additionalimage from instrument operations performed by the surgeon, based on thethree-dimensional coordinate information of the instrument tip outputfrom the instrument tip detecting unit 402, and the coordinateinformation of the region for each additional image that has been storedin the display region storage unit 401.

The operations of the 3D display device 20 according to the sixthmodification of the second embodiment are the same as those of the fifthmodification of the second embodiment illustrated in FIG. 29.

In a case where an instruction input to display an additional image isdetected in step S1110, the flow advances to step S1180.

The display control signal generating unit 405 detects input ofoperating the size of the additional image in step S1180. That is, thedisplay control signal generating unit 405 detects the passage detectingunit 403 outputting information of size specification input for aparticular additional image from instrument operations performed by thesurgeon, based on the three-dimensional coordinate information of theinstrument tip output from the instrument tip detecting unit 402, andcoordinate information of the region for each additional image that hasbeen stored in the display region storage unit 401.

In a case where input specifying the size of an additional image isdetected in step S1180, the flow advances to step S1190. In a case whereinput specifying the size of an additional image is not detected in stepS1180, the flow advances to step S1200.

Advantages

According to the sixth modification of the second embodiment describedabove, at the time of displaying an additional image over or adjacent toa 3D image from a stereo endoscope, the user, i.e., the surgeon, can usea surgical instrument in use which is in the images being shot by theendoscope camera. The display region of the additional image is decided,avoiding states with great difference in depth between the inside andoutside of the boundary of the display region of the additional image,and states with depth contradiction. This enables the additional imageto be displayed at a size which the user, i.e., the surgeon needs, whileavoiding depth contradictions such as an additional image which appearsto have embedded itself in the arms of the forceps, thereby alleviatingfatigue of the user, i.e., the surgeon. Further, the user can specifythe size of the additional image using the surgical instrument beingused for the surgery, without using a special interface device. That isto say, the surgeon can specify the size of the additional image withoutturning loose of the surgical instrument being used. Accordingly, the 3Ddisplay device can be operated without loosing efficiency in surgery.

Seventh Modification of Second Embodiment

Description has been made that the 3D display device 20 according to thesecond embodiment displays an additional image at an additional imagedisplay position stored in the additional image position storage unit140 which is also a position avoiding a display-forbidden region storedin the display-forbidden region storage unit 210, without the useroperating the position of the additional image. Description will be madein the present modification regarding a case where the user inputs adisplay position of the additional image using the operating input unit404, thereby instructing a display position.

The configuration of the 3D display device 20 according to a seventhmodification is the same as that of the sixth modification of the secondembodiment, except that the output of the display control signalgenerating unit 405 is input to the additional image position storageunit 140. Accordingly, illustration by way of drawings and descriptionthereof will be omitted.

FIG. 31 is a flowchart illustrating the operations of the 3D displaydevice 20 according to the seventh modification of the secondembodiment. FIG. 31 is the same as FIG. 8 in the second embodiment,other than step S1600 having been omitted and steps S1120, S1130, S1610,S1620, and S1630 having been added. Portions which are the same as thosein FIG. 8 will be denoted with the same reference numerals, anddescription will be omitted.

First, the endoscope camera 111 generates image information for 3Ddisplay having left and right disparity, as a main image, and the vitalsigns sensor 121 measures the current cardioelectric potential and bloodpressure of the patient as additional information (step S1100).

The display control signal generating unit 405 acquires operations whichthe user has made regarding the additional image at the operating inputunit 404, and detects instruction input to display an additional image(step S1110). In a case where additional image display instruction inputis detected in step S1110, i.e., in a case where step S1110 yields“yes”, the flow advances to step S1120. In a case where additional imagedisplay instruction input is not detected in step S1110, i.e., in a casewhere step S1110 yields “no”, the flow advances to step S1700.

In step S1700, the image compositing unit 180 composites an endoscopecamera image which is a main image with no additional image as a displayimage.

In step S1120, the display control signal generating unit 405 furtherdetects input of operating the position of the additional image. In acase where input specifying the position of the additional image isdetected in step S1120, i.e., in the case step S1120 yields “yes”, theflow advances to step S1130. In a case where input specifying theposition of the additional image is not detected in step S1120, i.e., inthe case step S1120 yields “no”, the flow advances to step S1200.

In step S1130, the display control signal generating unit 405 outputsthe information of position of the additional image in the instructioninput relating to display of the additional image acquired in step S1110to the additional image position storage unit 140. The additional imageposition storage unit 140 stores the information of the position of theadditional image which the display control signal generating unit 405has output (step S1130).

The display region candidate deciding unit 160 decides one candidate forthe display region, from the size of the additional image stored in theadditional image size storage unit 150, and the position of theadditional image stored in the additional image position storage unit140 (step S1200). For example, the display region candidate decidingunit 160 arbitrarily selects one unselected combination of additionalimage size stored in the additional image size storage unit 150 andadditional image position stored in the additional image positionstorage unit 140, and decides a region represented by the selectedcombination as a display region candidate.

Note that the newest information of the additional image positioninformation stored in the additional image position storage unit 140 isused to decide the display region candidate in step S1200. Accordingly,in a case where step S1130 has been executed, the position informationrecorded in step S1130 is used to decide the display region candidate.

The display region candidate deciding unit 160 determines whether or notthe display region candidate decided in step S1200 includes thedisplay-forbidden region stored in the display-forbidden region storageunit 210 (step S1210). In a case where determination is made in stepS1210 that the display region candidate includes the display-forbiddenregion, i.e., in a case where step S1210 yields “yes”, the flow advancesto step S1610.

In a case where determination is made in step S1210 that the displayregion candidate does not include the display-forbidden region, i.e., ina case where step S1210 yields “no”, the flow advances to step S1300.

The depth suitability determination unit 170 extracts a boundary line orboundary plane of the additional image display region candidate decidedin step S1200, and extracts depth information in the periphery of theboundary line or boundary plane (step S1300).

The depth suitability determination unit 170 further compares the depthof the display region of the additional image obtained from the positioninformation of the additional image stored in the additional imageposition storage unit 140 with the depth of the main image at theportion adjacent to the boundary line or boundary plane of theadditional image extracted in step S1300 (step S1400).

The depth suitability determination unit 170 determines whether or notthe difference in depth of the main image and additional image displayedacross the boundary line or boundary plane (step S1500). In a case wheredetermination is made in step S1500 that the difference in depth exceedsthe tolerance range, i.e., in a case where step S1500 yields “yes”, theflow advances to step S1610. In a case where determination is made instep S1500 that the difference in depth is within the tolerance range,i.e., in a case where step S1500 yields “no”, the flow advances to stepS1700.

In step S1700, the image compositing unit 180 composites the main imageacquired in step S1100 and the additional image representing theadditional information acquired in step S1100. Specifically, the imagecompositing unit 180 displays the main image on the screen of thedisplay 190, and displays the additional image so as to be displayed inthe display region which is the display region candidate decided in stepS1200, thus compositing the image.

The display 190 displays the 3D image composited in step S1700 (stepS1800), and subsequently the flow returns to step S1100.

In step S1610, the display region candidate deciding unit 160 changesthe size of the additional image, and stores the changed size in theadditional image size storage unit 150 (step S1610). For example, thedisplay region candidate deciding unit 160 reduces the size of theadditional image without changing the display position of the additionalimage so that the additional image display region does not include thedisplay-forbidden region, i.e., so that the additional image displayregion is outside of the display-forbidden region. Changing of size isperformed by maintaining the center-of-gravity position of theadditional image display region while reducing the additional image sothat the perimeter of the display region moves closer toward thecenter-of-gravity. Alternatively, one of the sides of the additionalimage display region that does not include the display-forbidden regionmay be fixed and the display region reduced in relation to theadditional image.

The display region candidate deciding unit 160 determines whether or notthe size of the additional image changed in step S1610 is within apredetermined certain range (step S1620). In a case where determinationis made in step S1620 that the size of the additional image is withinthe predetermined certain range, i.e., “yes” in step S1620, the flowreturns to step S1200. In a case where determination is made in stepS1620 that the size of the additional image is not within thepredetermined certain range, i.e., “no” in step S1620, the flow advancesto step S1630. For example, in a case where the additional image isrectangular, the predetermined size range of the additional image is 3cm to 10 cm in height, and 5 cm to 13 cm in width. In a case where thesize of the additional image reduced in step S1610 is within this range,e.g., the size of the additional image is 4 cm in height and 7 cm inwidth, the flow returns to step S1200. In a case where the size of theadditional image reduced in step S1610 is not within this range, e.g.,the size of the additional image is 2 cm in height and 4 cm in width,the flow advances to step S1630.

In step S1630, the display region candidate deciding unit 160 transmitsa signal to the image compositing unit 180 indicating that the size ofthe additional image is outside of the predetermined range. The imagecompositing unit 180 responds to this signal and generates an image of amessage prompting the user to re-input the display position of theadditional image. The display 190 then displays the message screen whichthe image compositing unit 180 has generated (step S1630). Afterexecuting step S1630, the flow returns to step S1110.

Repeating steps S1200 through S1610 causes the 3D display device 20 todecide display region candidates while avoiding the display-forbiddenregion. Further, repeating steps S1110 through S1630 causes the 3Ddisplay device 20 to decide display regions where the difference indepth within and outside of the boundary plane of the additional imagedoes not exceed the tolerance range.

The following methods may also be used as methods to input the positionof additional images using the operating input unit 404. For example,the user may select one display position from predetermined displaypositions. Alternatively, the user may specify the center-of-gravitypoint of the additional image display region, or the user may specifythe position of the sides of the additional image display region.

Advantages

According to the seventh modification of the second embodiment describedabove, at the time of displaying an additional image over or adjacent toa 3D image from a stereo endoscope, the user specifies the displayposition of the additional image. The 3D display device 20 decides thedisplay region of the additional image, avoiding states with greatdifference in depth between the inside and outside of the boundary ofthe display region of the additional image, and states with depthcontradiction, and also following the position specified by the user.This enables the additional image to be displayed at a position whichthe user, i.e., the surgeon needs, while avoiding depth contradictionssuch as an additional image which appears to have embedded itself in thearms of the forceps, thereby alleviating fatigue of the user, i.e., thesurgeon.

Eighth Modification of Second Embodiment

Description has been made that the 3D display device 20 according to thesecond embodiment displays an additional image at an additional imagedisplay position stored in the additional image position storage unit140 which is also a position avoiding a display-forbidden region storedin the display-forbidden region storage unit 210, without the useroperating the position of the additional image. Description will be madein the present modification regarding a case where the user inputs adisplay position and size of the additional image using the operatinginput unit 404, thereby instructing a display position and size. Notethat in a case where the display region of the additional image of whichthe display position and size is specified by user input includes thedisplay-forbidden region, and in a case where the difference in depthbetween within and outside of the boundary plane of the additional imagedisplay region exceeds the tolerance range, the 3D display deviceautomatically adjusts the position and size of the additional imagedisplay region. If the user is dissatisfied with the results ofautomatic adjustment, or of the display region obtained within the rangeof automatic adjustment is unsatisfactory, the 3D display device requestthe user to re-input the position and size of the additional imagedisplay region.

The configuration of the 3D display device 20 according to an eighthmodification is the same as that of the sixth modification of the secondembodiment, except that the output of the display control signalgenerating unit 405 is input to the additional image position storageunit 140 and the additional image size storage unit 150. Accordingly,illustration by way of drawings and description thereof will be omitted.

FIGS. 32A and 32B are a flowchart illustrating the operations of the 3Ddisplay device 20 according to the eighth modification.

First, the endoscope camera 111 acquires image information for 3Ddisplay having left and right disparity, as a main image, and the vitalsigns sensor 121 measures the current cardioelectric potential and bloodpressure of the patient as additional information (step S1100).

The display control signal generating unit 405 acquires operations whichthe user has made regarding the additional image at the operating inputunit 404, and detects instruction input to display an additional image(step S1110). In a case where additional image display instruction inputis detected in step S1110, i.e., in a case where step S1110 yields“yes”, the flow advances to step S1120. In a case where additional imagedisplay instruction input is not detected in step S1110, i.e., in a casewhere step S1110 yields “no”, the flow advances to step S1700.

In step S1700, the image compositing unit 180 composites an endoscopecamera image which is a main image with no additional image as a displayimage.

In step S1120, the display control signal generating unit 405 furtherdetects input of operating the position of the additional image. In acase where input specifying the position of the additional image isdetected in step S1120, i.e., in the case step S1120 yields “yes”, theflow advances to step S1130. In a case where input specifying theposition of the additional image is not detected in step S1120, i.e., inthe case step S1120 yields “no”, the flow advances to step S1180.

In step S1130, the display control signal generating unit 405 outputsthe information of position of the additional image in the instructioninput relating to display of the additional image acquired in step S1110to the additional image position storage unit 140. The additional imageposition storage unit 140 stores the information of the position of theadditional image which the display control signal generating unit 405has output (step S1130).

In step S1180, the display control signal generating unit 405 furtherdetects input of operating the size of the additional image. In a casewhere input specifying the size of the additional image is detected instep S1180, i.e., in the case step S1180 yields “yes”, the flow advancesto step S1190. In a case where input specifying the size of theadditional image is not detected in step S1180, i.e., in the case stepS1180 yields “no”, the flow advances to step S1200.

In step S1190, the display control signal generating unit 405 outputsthe information of size of the additional image in the instruction inputrelating to display of the additional image acquired in step S1110 tothe additional image size storage unit 150. The additional image sizestorage unit 150 stores the information of the size of the additionalimage which the display control signal generating unit 405 has output(step S1190).

The display region candidate deciding unit 160 decides one candidate forthe display region, from the size of the additional image stored in theadditional image size storage unit 150, and the position of theadditional image stored in the additional image position storage unit140 (step S1200). That is to say, the display region candidate decidingunit 160 decides as a display region candidate a region represented bythe combination of the size of the additional image stored in theadditional image size storage unit 150 and the position of theadditional image stored in the additional image position storage unit140.

Note that the newest information of the position of the additional imagestored in the additional image position storage unit 140, and the newestinformation of the size of the additional image stored in the additionalimage size storage unit 150, are used in step S1200. Accordingly, in acase where step S1130 has been executed, the position informationrecorded in step S1130 is used to decide the display region candidate.

The display region candidate deciding unit 160 determines whether or notthe display region candidate decided in step S1200 includes thedisplay-forbidden region stored in the display-forbidden region storageunit 210 (step S1210). In a case where determination is made in stepS1210 that the display region candidate includes the display-forbiddenregion, i.e., in a case where step S1210 yields “yes”, the flow advancesto step S1640.

In a case where determination is made in step S1210 that the displayregion candidate does not include the display-forbidden region, i.e., ina case where step S1210 yields “no”, the flow advances to step S1300.

The depth suitability determination unit 170 extracts a boundary line orboundary plane of the additional image display region candidate decidedin step S1200, and extracts depth information in the periphery of theboundary line or boundary plane (step S1300).

The depth suitability determination unit 170 further compares the depthof the display region of the additional image obtained by the displayregion candidate deciding unit 160 from the position information of theadditional image stored in the additional image position storage unit140 with the depth of the main image at the portion adjacent to theboundary line or boundary plane of the additional image extracted instep S1300 (step S1400).

The depth suitability determination unit 170 determines whether or notthe difference in depth of the main image and additional image displayedacross the boundary line or boundary plane exceeds the predeterminedtolerance range (step S1500). In a case where determination is made instep S1500 that the difference in depth exceeds the tolerance range,i.e., in a case where step S1500 yields “yes”, the flow advances to stepS1610. In a case where determination is made in step S1500 that thedifference in depth is within the tolerance range, i.e., in a case wherestep S1500 yields “no”, the flow advances to step S1700.

In step S1700, the image compositing unit 180 composites the main imageacquired in step S1100 and the additional image representing theadditional image information acquired in step S1100. Specifically, theimage compositing unit 180 displays the main image on the screen of thedisplay 190, and displays the additional image so as to be displayed inthe display region which is the display region candidate decided in stepS1200, thus compositing the image.

The display 190 displays the 3D image composited in step S1700 (stepS1800), and subsequently the flow returns to step S1100.

In step S1640, the display region candidate deciding unit 160 determineswhether or not the position and size of the additional image can beadjusted (step S1640). The method of determination in step S1640 will bedescribed later. In a case where determination is made in step S1640that the position and size of the additional image can be adjusted,i.e., step S1640 yields “yes”, the flow advances to step S1650. In acase where determination is made in step S1640 that the position andsize of the additional image cannot be adjusted, i.e., step S1640 yields“no”, the flow advances to step S1670.

In step S1650, the image compositing unit 180 composites a tentativedisplay image where the additional image display region output from thedepth suitability determination nit 170 has been overlaid on the mainimage, and the display 190 displays the tentative display imagecomposited by the image compositing unit 180 (step S1650). The actualadditional information may be displayed as the additional image in thetentative display image, or alternatively, a uniform plane image or afixed test image for tentative display of the region may be used.

The display control signal generating unit 405 acquires user operationsof the additional image made at the operating input unit 404 with regardto the tentative display of the additional image display regiondisplayed on the display 190 in step S1650, and detects instructioninput accepting the region of the additional image of which thetentative display position and size have been adjusted (step S1660). Ina case where instruction input accepting the tentatively displayedadditional image display region has been detected, i.e., step S1660yields “yes”, the flow advances to step S1700. In a case whereinstruction input accepting the tentatively displayed additional imagedisplay region is not detected, i.e., step S1660 yields “no”, the flowadvances to step S1670.

In step S1670 the image compositing unit 180 indicates to the user thatthe additional image cannot be displayed with the position and size ofthe additional image which the user has instructed, generates a messagescreen to prompt the user to re-specify the position and size of theadditional image, and the display 190 displays the message generated atthe image compositing unit 180 (step S1670). After displaying themessage screen on in step S1670, the flow returns to step S1110.

Repeating steps S1110 through S1630 causes the 3D display device 20 todecide display region candidates while avoiding the display-forbiddenregion, and to decide display region candidates where the difference indepth within and outside of the boundary plane of the additional imagedoes not exceed the tolerance range.

Details of Step S1640

FIG. 33 is a block diagram illustrating part of the 3D display device 20according to the eighth modification of the second embodiment in detail.The display region candidate deciding unit 160 includes a positionadjustment range storage unit 166, a size adjustment range storage unit162, a region deciding unit 163, a distance calculating unit 164, and aregion storage unit 165.

The position adjustment range storage unit 166 stores a predeterminedrange where an additional image can be moved, as a range where thedisplay position of an additional image which the user has specified canbe automatically adjusted. For example, this automatically-adjustablerange is a range 20% or less the length of the long side of theadditional image which the user has specified, in the long sidedirection or major axis direction, and 20% or less the length of theshort side in the short side direction or minor axis direction. Forexample, in a case where the additional image is a rectangle, thedisplay position of the additional image can be adjusted by movinghorizontally within a range of 20% or less of the long side, and bymoving vertically within a range of 20% or less of the short side. Thedisplay position of the additional image in the depth direction is notchanged here.

The size adjustment range storage unit 162 stores the predeterminedautomatically-adjustable range as a range where the display size of theadditional image specified user can be automatically adjusted. Forexample, this automatically-adjustable range is a range of 15% or lessincrease or decrease in the length of the major axis and minor axis ofthe additional image which the user has specified. For example, in acase where the additional image is a rectangle, the major axis and minoraxis can both be adjusted within a range of 85% to 115% as to the sizewhich the user has specified. Note however, that the scale of change isthe same for the long sides and the short side, i.e., deformation of theadditional image display region (size change with the aspect ratiochanges) is not performed here. Further, the size of the additionalimage in the depth direction is not changed here.

The region deciding unit 163 receives input from the additional imageposition storage unit 140, the additional image size storage unit 150,the size adjustment range storage unit 162, the distance calculatingunit 164, the region storage unit 165, and the depth suitabilitydetermination unit 170, and outputs information relating to displayregion candidates for the additional image to the distance calculatingunit 164 and the region storage unit 165. The region deciding unit 163acquires the display position of the additional image and the size ofthe additional image which the user has specified, from the additionalimage position storage unit 140 and additional image size storage unit150, and decides the display region of the additional image. The regiondeciding unit 163 further follows input from the distance calculatingunit 164 or input from the depth suitability determination unit 170 andadjusts the position and size of the display region of the additionalimage, based on the adjustment range of the additional image displayposition stored in the position adjustment range storage unit 166 andthe adjustment range of the additional image display size stored in thesize adjustment range storage unit 162.

The distance calculating unit 164 calculates the distance between theadditional image display region candidate acquired by the regiondeciding unit 163, and the display-forbidden region stored in thedisplay-forbidden region storage unit 210, in the image display space.For example, the distance calculating unit 164 maps each of thedisplay-forbidden region and the additional image display regioncandidate in the image display space, onto the display plane, andcompare the positions of the two on the display plane, therebycalculating the distance between the two. The distance calculating unit164 can calculate the distance between the two regions from thedifference in x coordinates and y coordinates in the coordinate systemsuch as illustrated in FIG. 2B, for example. The distance calculatingunit 164 outputs the calculation results to the region deciding unit163.

The region storage unit 165 stores information relating to theadditional image display region candidates decided by the regiondeciding unit 163.

FIG. 34 is a flowchart illustrating detailed operations of step S1640 inthe operations of the 3D display device 20 in the eighth embodiment ofthe second embodiment. In step S1640, the display region candidatedeciding unit 160 determines whether or not the position and the size ofthe additional image can be adjusted.

The distance calculating unit 164 calculates the distance between thenewest additional image display region candidate decided at the regiondeciding unit 163 and the display-forbidden region stored in thedisplay-forbidden region storage unit 210 (step S1641). The distance isexpressed by a positive value if the two regions mapped to the displayplane are separated, and a negative value if two regions areoverlapping. In a case where there is no overlapping between the tworegions, the distance between the regions is the distance between thetwo closest points of the two regions. In a case where the two regionsare overlapping, the distance is the negative value of the longest linethat can be fit into the overlapped region.

Based on the distance between regions measured in step S1641, the regiondeciding unit 163 determines whether or not the newest additional imagedisplay region candidate includes the display-forbidden region stored inthe display-forbidden region storage unit 210 (step S1642). If thedistance is a negative value, this means that the two regions areoverlapped, and consequently the additional image display regioncandidate includes the display-forbidden region. In a case wheredetermination is made in step S1642 that the additional image displayregion candidate includes the display-forbidden region, i.e., step S1642yields “yes”, the flow advances to step S1643. In a case wheredetermination is made in step S1642 that the additional image displayregion candidate does not include the display-forbidden region, i.e.,step S1642 yields “no”, the flow advances to step S1647.

In step S1643, the region deciding unit 163 determines whether or notthe display region candidate can be moved to a position not includingthe display-forbidden region, based on the negative inter-regiondistance calculated in step S1641, i.e., the overlapping distance of theregions, and the adjustment range of the position stored in the positionadjustment range storage unit 166. This determination is made by whetheror not the absolute value of the inter-region distance is within theposition adjustment range, for example. In a case where determination ismade in step S1643 that the display region candidate can be moved to aposition not including the display-forbidden region, i.e., step S1643yields “yes”, the flow advances to step S1644. In a case wheredetermination is made in step S1643 that the display region candidatecannot be moved to a position not including the display-forbiddenregion, i.e., step S1643 yields “no”, the flow advances to step S1645.

In step S1644, the region deciding unit 163 moves this additional imagedisplay region candidate, and resets this position where the displayregion candidate does not include the display-forbidden region as theposition of the display region candidate. After executing step S1644,the flow returns to step S1641.

In step S1645, the region deciding unit 163 determines whether or notthe size of the display region candidate can be adjusted so that thedisplay region candidate does not include the display-forbidden region,based on the negative inter-region distance calculated in step S1641,i.e., the overlapping distance of the regions, and the size adjustmentrange stored in the size adjustment range storage unit 162. Thisdetermination is made by whether or not the absolute value of theinter-region distance is within the length adjustment range, forexample. For the additional image display region candidate to be made tonot include the display-forbidden region without changing the positionthereof, the size of the display region candidate is reduced withoutchanging the center-of-gravity position of the display region candidateserving as a position reference, for example. In a case wheredetermination is made in step S1645 that the size of the display regioncandidate can be adjusted so that the display region candidate does notinclude the display-forbidden region, i.e., step S1645 yields “yes”, theflow advances to step S1646. In a case where determination is made instep S1645 that the size of the display region candidate cannot beadjusted so that the display region candidate does not include thedisplay-forbidden region, i.e., step S1645 yields “no”, the flowadvances to step S1670.

In step S1646, the region deciding unit 163 changes the size of thisadditional image display region candidate, and resets this size wherethe display region candidate does not include the display-forbiddenregion as the size of the display region candidate. After executing stepS1646, the flow returns to step S1641.

Repeating steps S1641 through S1644 or 1646 adjusts the position or sizewithin the predetermined adjustment range, thereby deciding anadditional image display region candidate not including thedisplay-forbidden region. In a case where an additional image displayregion candidate not including the display-forbidden region cannot bedecided by adjusting the position or size within the predeterminedadjustment range, the image compositing unit 180 presents the user is arequest to re-specify the position and size of the display region of theadditional image (step S1670). For example, the image compositing unit180 displays a request message on the screen of the display 190 forre-specifying.

On the other hand, in step S1647 the depth suitability determinationunit 170 determines whether or not the difference between the depthwithin and outside of the boundary plane of the additional image displayregion candidate decided by the region deciding unit 163 exceeds thetolerance range. The operations of step S1647 are the same as theoperations of steps S1300 through S1500. In a case where determinationis made in step S1647 that the difference between the depth within andoutside of the boundary plane exceeds the tolerance range, i.e., “yes”in step S1647, the flow advances to step S1648. In a case wheredetermination is made in step S1647 that the difference between thedepth within and outside of the boundary plane does not exceed thetolerance range, i.e., “no” in step S1647, the flow advances to stepS1650.

In step S1648, the region deciding unit 163 determines whether or notthe position of the additional image display region candidate can bemoved within the position adjustment range stored in the positionadjustment range storage unit 166. That is to say, the region decidingunit 163 determines whether or not there is a possibility of movementnot exceeding the adjustment range of the additional image displayregion stored in the position adjustment range storage unit 166, e.g.,the adjustment range which is 20% of the length of the additional imageregion in the movement direction from the specified display position. Anexample will be considered here where the additional image isrectangular, there are four selections which can be made from thespecified display position in the horizontal direction, which aremovement of 10% of the horizontal axis to the right, movement of 20% tothe right, movement of 10% of the horizontal axis to the left, andmovement of 20% to the left, and there are four selections which can bemade from the specified display position in the vertical direction,which are movement of 10% of the vertical axis upwards, movement of 20%upwards, movement of 10% of the vertical axis downwards, and movement of20% downwards. In this case, if there is any combination of the fourtypes of movement in the horizontal direction and the four types ofmovement in the vertical direction that has not been subjected toposition adjustment operations, the region deciding unit 163 determinesthat the position of the display region candidate can be moved. On theother hand, in a case that all combinations have been subjected toposition adjustment operations, the region deciding unit 163 determinesthat the position of the display region candidate cannot be moved.

In a case where determination is made in step S1648 that the position ofthe display region candidate can be moved, i.e., step S1648 yields“yes”, the flow advances to step S1649. In a case where determination ismade in step S1648 that the position of the display region candidatecannot be moved, i.e., step S1648 yields “no”, the flow advances to stepS1901.

In step S1649, the region deciding unit 163 selects one of theselectable display positions of the additional image display regioncandidates, and moves the display position of the additional imagedisplay region candidate (step S1649). After executing step S1649, theflow returns to step S1641.

In step S1901, the region deciding unit 163 determines whether or notthe size of the additional image display region candidate can be changedwithin the size adjustment range stored in the size adjustment rangestorage unit 162. That is to say, the region deciding unit 163determines whether or not the size of the additional image displayregion specified by user input in step S1120 can be changed within theadjustment range of the additional image display region stored in thesize adjustment range storage unit 162. For example, determination ismade regarding whether or not there is a possibility of extending orcompressing the size of the additional image display region specified byuser input within 15% in the major axis direction or minor axisdirection of the display region. An example will be considered herewhere the additional image is rectangular, and can be expanded by 5%,expanded by 10%, expanded by 15%, compressed by 5%, compressed by 10%,or compressed by 15%, in length of the additional image rather than areaof the additional image. If there are any of the six types of sizechange remaining as an option, the region deciding unit 163 determinesthat the size of the display region candidate can be changed. If all ofthe six types of size change have already been used for adjustmentoperations, the region deciding unit 163 determines that the size of thedisplay region candidate cannot be changed.

In a case where determination is made in step S1901 that the size of thedisplay region candidate can be changed, i.e., step S1901 yields “yes”,the flow advances to step S1902. In a case where determination is madein step S1901 that the size of the additional image display regioncandidate cannot be changed, i.e., step S1901 yields “no”, the flowadvances to step S1670.

In step S1902, the region deciding unit 163 selects one of theselectable sizes of the additional image display region candidate, andchanges the size of the additional image display region candidate (stepS1902). After executing step S1902, the flow returns to step S1641.

Repeating steps S1641 through S1649 or S1902 enables the difference indepth within and outside of the boundary plane of the additional imageto be adjusted to within the tolerance range. In a case where thedifference in depth within and outside of the boundary plane of theadditional image cannot be adjusted to be within the tolerance range byposition and size adjustment within the predetermined adjustment range,the image compositing unit 180 presents the user is a request tore-specify the position and size of the additional image display region(step S1670). For example, the image compositing unit 180 displays arequest message for re-specifying on the screen of the display 190.

While description has been made in the eighth modification of the secondembodiment that the display region candidate deciding unit 160 does notadjust the display position in the depth direction, an arrangement maybe made where the display position is moved in the depth position in thesame way as the horizontal direction and vertical direction to performadjustment. Particularly, in a case where there is difference in depthat the boundary plane of the additional image which slightly exceeds thetolerance range, there may be cases where moving the display region ofthe additional image in the depth direction is effective. In this case,the additional image can be moved toward the near side and the far sideto adjust the depth direction, with the range of adjustment in the depthdirection begin set to 10% or less of the original depth range, or setto 10% or less of the distance to the display plane from the originalaverage depth position or smallest depth position.

Advantages

According to the eighth modification of the second embodiment describedabove, at the time of the 3D display device 20 displaying an additionalimage over or adjacent to a 3D image from a stereo endoscope, the userspecifies the display position of the additional image. The 3D displaydevice 20 decides the display region of the additional image, avoidingstates with great difference in depth between the inside and outside ofthe boundary of the display region of the additional image, and stateswith depth contradiction, and also following the position specified bythe user. This enables the additional image to be displayed at aposition which the user, i.e., the surgeon needs, while avoiding depthcontradictions such as an additional image which appears to haveembedded itself in the arms of the forceps, thereby alleviating fatigueof the user, i.e., the surgeon.

Ninth Modification of Second Embodiment

Description has been made regarding the fifth through eighthmodifications of the second embodiment that the difference in depthwithin and outside of the boundary plane of the additional image is keptfrom exceeding the tolerance range, by adjusting the position and sizeof the additional image display region which the user has specified.However, deformation of the additional image display region is notperformed in the fifth through eighth modifications of the secondembodiment. In the ninth embodiment, the difference in depth within andoutside of the boundary plane of the additional image is prevented fromexceeding the tolerance range, by deforming the additional image displayregion. The configuration of the display of additional information ischanged in the ninth embodiment in accordance with deformation of theadditional image display region. In a case where the additionalinformation is information of a type which the spatial configurationcannot be deformed, such as image information, deformation of thedisplay region is not valid. Description will be made in the presentmodification regarding deformation of the display region involvingchanging of the configuration of the display, using an example of vitalsigns information.

Configuration

FIG. 35 is a block diagram illustrating a functional configuration ofthe 3D display device 20 according to the ninth modification of thesecond embodiment. The 3D display device 20 in FIG. 35 is the same asthe 3D display device 20 in FIG. 7, except for the points that thedisplay region candidate deciding unit 160 has been replaced with adisplay region candidate deciding unit 360, an additional image shapestorage unit 310 has been added, a display control signal generatingunit 220 and operating input unit 221 have been added as the input unit200, the output of the display control signal generating unit 220 isinput to the additional image position storage unit 140, and the outputof the depth information generating unit 130 is input to the displayregion candidate deciding unit 360. Processing units which are the sameas those in FIG. 8 are denoted with the same reference numerals, anddescription thereof will be omitted.

The 3D display device 20 includes the endoscope camera 111, vital signssensor 121, MRI image storage unit 122, 3D image compositing unit 123,depth information generating unit 130, additional image position storageunit 140, additional image size storage unit 150, additional image shapestorage unit 310, display region candidate deciding unit 360, depthsuitability determination unit 170, image compositing unit 180, display190, display-forbidden region storage unit 210, and input unit 200. Theinput unit 200 includes the display control signal generating unit 220and the operating input unit 221.

The additional image shape storage unit 310 stores the shape of theadditional image display region. In a case where the display region ofthe additional image is a rectangle, the additional image shape storageunit 310 stores the ratio of the vertical and horizontal sides. Inanother case, if the additional image display region is an ellipse, theadditional image shape storage unit 310 stores the ratio of the majoraxis and minor axis, or the like, thus storing information by which theshape of the additional image can be decided.

The display region candidate deciding unit 360 decides the position,size, and shape of the additional image display region candidate byreferencing information stored in the additional image position storageunit 140, additional image size storage unit 150, additional image shapestorage unit 310, and display-forbidden region storage unit 210, andfurther acquires depth information, generated by the depth informationgenerating unit 130, of the image acquired by the endoscope camera 111which is the main image.

Operations

FIG. 36 is a flowchart illustrating the processing operations of the 3Ddisplay device 20 according to the ninth modification of the secondembodiment. FIG. 36 is the same as FIG. 31, other than step S1620 havingbeen omitted from step S1600, and step S2010 having been added, in theprocessing operation flowchart of the 3D display device 20. Portionswhich are the same as those in FIG. 31 will be denoted with the samereference numerals, and description will be omitted. The processingoperations of the 3D display device 20 according to the ninthmodification of the second embodiment will be described below followingFIG. 36.

First, the endoscope camera 111 acquires image information for 3Ddisplay having left and right disparity, as a main image, and the vitalsigns sensor 121 measures the current cardioelectric potential and bloodpressure of the patient as additional information (step S1100).

The display control signal generating unit 220 acquires operations whichthe user has made regarding the additional image at the operating inputunit 221, and detects instruction input to display an additional image(step S1110). In a case where additional image display instruction inputis detected in step S1110, i.e., in a case where step S1110 yields“yes”, the flow advances to step S1120. In a case where additional imagedisplay instruction input is not detected in step S1110, i.e., in a casewhere step S1110 yields “no”, the flow advances to step S1700.

In step S1700, the image compositing unit 180 composites an endoscopecamera image which is a main image with no additional image as a displayimage.

In step S1120, the display control signal generating unit 220 furtherdetects input of operating the position of the additional image. In acase where input specifying the position of the additional image isdetected in step S1120, i.e., in the case step S1120 yields “yes”, theflow advances to step S1130. In a case where input specifying theposition of the additional image is not detected in step S1120, i.e., inthe case step S1120 yields “no”, the flow advances to step S1200.

In step S1130, the display control signal generating unit 220 outputsthe information of position of the additional image in the instructioninput relating to display of the additional image acquired in step S1110to the additional image position storage unit 140. The additional imageposition storage unit 140 stores the information of the position of theadditional image which the display control signal generating unit 220has output (step S1130).

The display region candidate deciding unit 360 decides one candidate forthe display region, from the size of the additional image stored in theadditional image size storage unit 150, and the position of theadditional image stored in the additional image position storage unit140 (step S1200). For example, the display region candidate decidingunit 360 arbitrarily selects one unselected combination of additionalimage size stored in the additional image size storage unit 150, anddecides a region represented by a combination thereof with the selectedadditional image position stored in the additional image positionstorage unit 140, as a display region candidate.

Note that the newest information of the additional image informationstored in the additional image position storage unit 140 is used todecide the display region in step S1200. Accordingly, in a case wherestep S1130 has been executed, the size information recorded in stepS1130 is used to decide the display region candidate.

The display region candidate deciding unit 360 determines whether or notthe display region candidate decided in step S1200 includes thedisplay-forbidden region stored in the display-forbidden region storageunit 210 (step S1210). In a case where determination is made in stepS1210 that the display region candidate includes the display-forbiddenregion, i.e., in a case where step S1210 yields “yes”, the flow advancesto step S2010.

In a case where determination is made in step S1210 that the displayregion candidate does not include the display-forbidden region, i.e., ina case where step S1210 yields “no”, the flow advances to step S1300.

The depth suitability determination unit 170 extracts a boundary line orboundary plane of the additional image display region candidate decidedin step S1200, and extracts depth information in the periphery of theboundary line or boundary plane (step S1300).

The depth suitability determination unit 170 further compares the depthof the display region of the additional image obtained from the positioninformation of the additional image stored in the additional imageposition storage unit 140 by the display region candidate deciding unit360, with the depth of the main image at the portion adjacent to theboundary line or boundary plane of the additional image extracted instep S1300 (step S1400).

The depth suitability determination unit 170 determines whether or notthe difference in depth of the main image and additional image displayedacross the boundary line or boundary plane exceeds the predeterminedtolerance range (step S1500). In a case where determination is made instep S1500 that the difference in depth exceeds the tolerance range,i.e., in a case where step S1500 yields “yes”, the flow advances to stepS2010. In a case where determination is made in step S1500 that thedifference in depth is within the tolerance range, i.e., in a case wherestep S1500 yields “no”, the flow advances to step S1700.

In step S1700, the image compositing unit 180 composites the main imageacquired in step S1100 and the additional image representing theadditional information acquired in step S1100. Specifically, the imagecompositing unit 180 displays the main image on the screen of thedisplay 190, and displays the additional image so as to be displayed inthe display region on the main image which is the display regioncandidate decided in step S1200, thus compositing the image.

The display 190 displays the 3D image composited in step S1700 (stepS1800), and subsequently the flow returns to step S1100.

In step S2010, the display region candidate deciding unit 360 deformsthe shape of the additional image, and stores the deformed shape in theadditional image shape storage unit 310. For example, the display regioncandidate deciding unit 360 deforms the shape of the additional imagewithout changing the display position of the additional image, such thatthe additional image display region does not include thedisplay-forbidden region, i.e., so that the additional image displayregion is outside of the display-forbidden region. For example, in acase where the additional image display region is rectangular, and theupper portion of the additional image display region is overlapping thedisplay-forbidden region, the sides of the rectangle in the heightdirection are shortened, and the sides in the width direction arelengthened, thus deforming the display region into an even morehorizontally long shape, so that the additional image does not includethe display-forbidden region. Details of the deforming processing of theadditional image display region (step S2010) will be described later.After executing step S2010, the flow returns to step S1200.

Repeating steps S1200 through S2010 causes the 3D display device 20 todecide display region candidates while avoiding the display-forbiddenregion, and further to decide an additional image display region wherethe difference in depth does not exceed the tolerance range.

FIG. 37 is a functional block diagram illustrating part of the 3Ddisplay device 20 according to the ninth modification of the secondembodiment in detail. The display region candidate deciding unit 360includes a region deciding unit 363, the distance calculating unit 164,the region storage unit 165, a shape deciding unit 361, and a layoutdeciding unit 362.

The region deciding unit 363 accepts input from the additional imageposition storage unit 140, additional image size storage unit 150, shapedeciding unit 361, distance calculating unit 164, region storage unit165, layout deciding unit 362, and depth suitability determination unit170, and outputs information relating to a display region candidate ofthe additional image to the distance calculating unit 164 and regionstorage unit 165. The region deciding unit 363 acquires the displayposition of the additional image which the user has specified from theadditional image position storage unit 140, and acquires thepredetermined size of the additional image from the additional imagesize storage unit 150. The region deciding unit 363 decides the regionwhich the acquired display position and size indicates as being thedisplay region of the additional image. The region deciding unit 363further follows the input from the distance calculating unit 164, or theinput from the depth suitability determination unit 170 and layoutdeciding unit 362, to deform the display region of the additional image,and adjusts the display layout of the additional information within thedeformed display region.

The distance calculating unit 164 calculates the distance between theadditional image display region candidate acquired by the regiondeciding unit 363 and the display-forbidden region stored in thedisplay-forbidden region storage unit 210, in the image display space.For example, the distance calculating unit 164 maps each of thedisplay-forbidden region and the additional image display regioncandidate in the image display space, onto the display plane, andcompare the positions of the two on the display plane, therebycalculating the distance between the two. The distance calculating unit164 can calculate the distance between the two regions from thedifference in x coordinates and y coordinates on the x-y plane in thecoordinate system such as illustrated in FIG. 2B, for example. Thedistance calculating unit 164 outputs the calculation results to theregion deciding unit 363.

The region storage unit 165 stores information relating to theadditional image display region candidates decided by the regiondeciding unit 363.

The shape deciding unit 361 decides the shape and size of the additionalimage display region candidate, based on the distance between thedisplay-forbidden region and the additional image display regioncandidate obtained by the distance calculating unit 164, the depthinformation of the 3D image imaged by the endoscope camera 111, which isthe main image, acquired from the depth information generating unit 130,the shape information of the additional image stored in the additionalimage shape storage unit 310, the position of the additional imagestored in the additional image position storage unit 140, and the sizeof the additional image stored in the additional image size storage unit150. The shape deciding unit 361 outputs the shape and size of thedisplay region candidate that has been decided to the layout decidingunit 362 and the region deciding unit 363.

The layout deciding unit 362 decides a layout for displaying vital signsinformation within the additional image display region, according to theshape and size of the display region candidate of the additional imageacquired by the shape deciding unit 361. FIG. 38A is a schematic diagramillustrating an example of displaying vital signs information in astandard shape additional image display region. FIG. 38B is a schematicdiagram illustrating an example of displaying vital signs information inan additional image display region which is horizontally longer than thestandard shape. FIG. 38C is a schematic diagram illustrating an exampleof displaying vital signs information in two additional image displayregions for blood pressure and an electrocardiogram. In a case where thevital signs information are blood pressure and an electrocardiogram, thehorizontal axes represents time in both graphs, and in the graph forblood pressure the vertical axis represents pressure while in the graphfor the electrocardiogram the vertical axis represents potential. Therectangular display region candidate in FIG. 38A, which is the standardshape, has the two graphs arrayed vertically with the horizontal axesmatched, so time temporal synchronization is readily understood. On theother hand, the display region candidate illustrated in FIG. 38B whichis long in the horizontal direction allows the two graphs to be observedwithout vertical compression even for regions where the height is short,by arraying the two graphs horizontally. The arrangement in FIG. 38Cdisplays the two graphs divided into two regions, so the area of eachregion is small, the degree of freedom of the additional image displaypositions is high, and the graphs can be readily viewed withoutexcessive compression.

FIG. 39 is a flowchart illustrating part of the operations of the 3Ddisplay device 20 according to the ninth modification of the secondembodiment. The operations of step S2010 will be described withreference to FIG. 39.

In a case where determination is made in step S2010 in FIG. 36 that thedisplay region candidate includes the display-forbidden region, i.e.,step S1210 yields “yes”, or determination is made in step S1500 that thedifference in depth is not within the tolerance range i.e., step S1500yields “yes”, the display region candidate deciding unit 360 performsthe operations of step S2010.

First, the shape deciding unit 361 extracts a portion of the image wherethe amount of protrusion from the screen is great, based on the depthinformation of the main image acquired from the depth informationgenerating unit 130 (step S2011). Coordinates are set indicating theposition of the object in the image as illustrated in FIG. 2B. In a casewhere a positive value on the depth direction on the z axis representsthe direction of protruding from the screen and a negative valuerepresents the far side of the screen, the shape deciding unit 361extracts an object portion of which the value on the z axis exceeds 50cm as being a portion where the amount of protrusion is great.

The shape deciding unit 361 calculates the distance between the portionwhere the amount of protrusion is great that has been extracted in stepS2011, and the display region candidate of the additional image (stepS2012). For example, the shape deciding unit 361 maps the portion wherethe amount of protrusion is great and the display region of theadditional image on the coordinates x-y plane in FIG. 2B, i.e., on thedisplay plane, and obtains the distance on this plane where the tworegions are the closest.

The shape deciding unit 361 determines whether or not to shield theportion where the amount of protrusion is great by the additional image,based on the shape of the portion where the amount of protrusion isgreat that has been obtained in step S2011, and the distance obtained instep S2012 (step S2013). Examples of cases where determination is madeby the shape deciding unit 361 to shield the portion where the amount ofprotrusion is great by the additional image include a case where themapped portion to the x-y plane of the portion where the amount ofprotrusion is great is continuous and the area of the mapped portion issmaller than a predetermined value, a case the mapped portion to the x-yplane of the portion where the amount of protrusion is great is notcontinuous but the distribution range overlaps any continuous area ofthe predetermined area, and a case where the portion where the amount ofprotrusion is great that has been mapped to the x-y plane and theadditional image display region candidate mapped to the x-y planeoverlap. In a case where determination is made in step S2013 to shieldthe portion where the amount of protrusion is great by the additionalimage, i.e., step S2013 yields “yes”, the flow advances to step S2014.

In step S2014, the shape deciding unit 361 deforms the region of theadditional image so as to be a shape and size including the protrudingportion to be shielded. The additional image is a rectangle here.Accordingly, the shape deciding unit 361 takes the distance in thehorizontal direction, between the right edge and left edge in the x axisdirection in FIG. 2B, of the portion to be shielded on the displayplane, i.e., the shape mapped to the x-y plane in FIG. 2B, for example.In the same way, the shape deciding unit 361 takes the distance in thevertical direction, between the top edge and bottom edge in the y axisdirection in FIG. 2B, of the portion to be shielded. The shape decidingunit 361 then uses these distances as the length of the height and widthsides of the additional image display region following deformation, thusyielding the shape and size thereof.

In a case where determination is made in step S2013 not to shield theportion where the amount of protrusion is great by the additional image,i.e., step S2013 yields “no”, the flow advances to step S2015.

In step S2015, the shape deciding unit 361 determines whether or notthis additional image display region candidate includes thedisplay-forbidden region, based on this additional image display regioncandidate calculated by the distance calculating unit 164 and thedistance to the display-forbidden region (step S2015). In a case wherethe additional image display region candidate is determined in stepS2015 to include the display-forbidden region, i.e., step S2015 yields“yes”, the flow advances to step S2016. In a case where the additionalimage display region candidate is determined in step S2015 to notinclude the display-forbidden region, i.e., step S2015 yields “no”, theflow advances to step S2017.

In step S2016, the shape deciding unit 361 deforms the shape of theadditional image display region candidate, in accordance with thedistance between this additional image display region candidate and thedisplay-forbidden region calculated by the distance calculating unit164. The shape deciding unit 361 performs deformation by changing thelength of the long sides and short sides so that the overlapping betweenthe additional image display region and the display-forbidden region isgone, in which the amount of change in the center-of-gravity position ofthe additional image display region due to the deformation of theadditional image display region is minimal.

In step S2017, the shape deciding unit 361 deforms the additional imagedisplay region candidate in accordance with the distance between theprotruding portion calculated in step S2012 and the additional imagedisplay region candidate. For example, the shape deciding unit 361changes the lengths of the long sides and short sides so that the sum ofthe distance between all protruding portions adjacent to or overlappingthis additional image display region candidate and the this additionalimage display region candidate is maximal. Alternatively, the shapedeciding unit 361 may perform deformation by changing the length of thelong sides and short sides to resolve overlapping with protrudingportions, in which the amount of change in the center-of-gravityposition of the additional image display region due to the deformationof the additional image display region is minimal.

The layout deciding unit 362 changes the display layout of theadditional information in accordance with the aspect ratio of theadditional image display region candidate deformed in step S2014, stepS2016, or step S2017 (step S2018). For example, in a case where theshape of the additional image display region candidate is a rectangle,and the height of the display region is smaller than 8 cm, the layoutdeciding unit 362 changes the layout to that where multiple graphshaving time as the horizontal axis are arrayed horizontally, asillustrated in FIG. 38B. In a case where the height of the displayregion is 8 cm or longer, the layout deciding unit 362 changes thelayout to that where the time axes of the two graphs are matched andarrayed vertically, as illustrated in FIG. 38A.

Description has been made above that in a case where determination ismade in step S2015 that the additional image display region includes thedisplay-forbidden region, i.e., step S2015 yields “yes”, in step S2016the shape deciding unit 361 deforms the shape of the display regioncandidate so as to avoid the display-forbidden region. However, in acase where such deformation would result in the area of the displayregion candidate being extremely small, equal to or smaller than apredetermined threshold value, the shape deciding unit 361 may transmita signal to the image compositing unit 180 indicating that the area isextremely small. The image compositing unit 180 responds to the receivedsignal by displaying a message on the display 190 prompting the user tore-specify the display position of the additional image.

As described above, the 3D display device 20 according to the ninthmodification of the second embodiment changes the shape of theadditional image display region and the layout of the additionalinformation within the display region when the user has specified adisplay position of the additional image when superimposing theadditional image on the 3D image imaged by a stereo endoscope camera.Accordingly, the 3D display device 20 decides the display region of theadditional image, avoiding states with great difference in depth betweenthe inside and outside of the boundary of the display region of theadditional image, and states with depth contradiction, while alsogenerally following the position specified by the user. Also, shieldingthe main image by the additional image enables avoiding states withgreat difference in depth between the inside and outside of the boundaryof the display region of the additional image, and states with depthcontradiction. This enables the additional image to be displayed at aposition which the user, i.e., the surgeon needs, while avoiding depthcontradictions such as an additional image which appears to haveembedded itself in the arms of the forceps, thereby alleviating fatigueof the user, i.e., the surgeon.

Also note that while description has been made in the ninth modificationof the second embodiment that the display region candidate deciding unit360 changes the layout of the biological data graphs in a case wheredetermination is made that the difference in depth exceeds the tolerancerange even if the depth suitability determination unit 170 changes theposition of the candidate region, an arrangement may be made where thelayout of the biological data graphs is changed in a case wheredetermination is made that the difference in depth exceeds the tolerancerange even if the depth suitability determination unit 170 changes thesize of the candidate region.

Description has been made in the ninth modification of the secondembodiment that the display region of the additional image is deformedsuch that the additional image shields a portion in the main image wherethe amount of protrusion is great. Another arrangement which may be madeis to change the depth for display of the additional image at the sametime as deforming the display region, adjusting the additional imagedisplay region so as to minimize the difference in depth between theadditional image and the main image at the boundary plane between theadditional image and the boundary plane. The phrase “minimize thedifference in depth” means to minimize the average difference in depth,or to place the depth position of the additional image at the greatestdepth position in the main image side.

Note that in the fifth through ninth modifications of the secondembodiment, the user may select one display region candidate out of themultiple display region candidates decided by the display regioncandidate deciding unit 160 or the display region candidate decidingunit 360. For example, processing may be added where an additional imageis displayed at all display region candidates, and the user uses theinput unit 200 to select a particular display region. FIG. 40A is adiagram illustrating a display example where four additional imagedisplay region candidates, set by the display region candidate decidingunit 160 or the display region candidate deciding unit 360, aredisplayed on the screen. In this example, the display region candidatesfor the additional image are set at positions in contact with the fourcorners of the display. Each additional image region has the additionalimage positions fixed to the corners of the display, and the sizeadjusted so as to not include a region stored in the display-forbiddenregion storage unit 210 and for the difference in depth within andoutside of the additional image display regions to be within thetolerance range, using similar operations to steps S1210 through S1610in FIG. 31 according to the seventh modification of the secondembodiment. The size-adjusted additional image display region candidatesare all displayed on the display as illustrated in FIG. 40A, and theuser selects a display region which has the most desirable position andsize.

FIG. 40B is an example of an operating input unit 221 for the user toselect a display region candidate. The operating input unit 221 in theexample in FIG. 40B has four buttons corresponding to positions on thedisplay, and a D-pad. The user can select press a button correspondingto a display region of which the position and size is most desirablefrom the four additional image display region candidates, or can use theD-pad to select one of the four additional image display regioncandidates using the D-pad.

While description has been made in the first through ninth modificationsof the second embodiment that the stereo endoscope camera images andvital signs information are acquired and displayed in real-time, anarrangement may be made where images imaged by the stereo endoscopecamera and recorded are displayed along with vital signs informationacquired and recorded synchronously with the images. In this case, theendoscope camera 111 and vital signs sensor 121 are replaced by an imagestorage unit and vital signs data storage unit. In a case where imagesrecorded beforehand are to be used, the depth information may begenerated beforehand, and stored as data synchronized with the imageinformation.

Third Embodiment

In the second embodiment of the present invention and the modificationsthereof, the 3D display device 20 has been described as performingreal-time display of images during endoscope surgery, taken with thestereo endoscope camera 111. In a third embodiment, images that havebeen taken with a stereo endoscope camera and stored are displayed asthe main image. Information recorded at the same time as imaging withthe stereo endoscope camera during surgery is stored as additionalinformation that has been temporally synchronized with the stereoendoscope camera images.

Configuration

FIG. 41 is a block diagram illustrating a functional configuration of a3D display device 30 according to the third embodiment. Theconfiguration is the same as the 3D display device 10 illustrated inFIG. 1, other than the points that the main image acquisition unit 110has been replaced by an endoscope image storage unit 112, the additionalinformation acquisition unit 120 has been replaced by a vital signsstorage unit 125, the MRI image storage unit 122, and the 3D imagecompositing unit 123, and a suitability determination result storageunit 340, optimal display plan deciding unit 320, and additional imagedisplay plan storage unit 330 have been newly added. Other componentsare the same as those of the 3D display device 20 in FIG. 1. Portionswhich are the same as those in FIG. 1 are denoted by the same referencenumerals, and description will be omitted.

The 3D display device 30 includes the endoscope image storage unit 112,vital signs storage unit 125, MRI image storage unit 122, 3D imagecompositing unit 123, depth information generating unit 130, additionalimage position storage unit 140, additional image size storage unit 150,display region candidate deciding unit 160, depth suitabilitydetermination unit 170, suitability determination result storage unit340, optimal display plan deciding unit 320, additional image displayplan storage unit 330, image compositing unit 180, and display 190.

The endoscope image storage unit 112 stores images of surgery using astereo endoscope. The images are stereo 3D images where images of thestereo endoscope camera used in the surgery are stored as moving images,more particularly as a right-eye image and left-eye image which havebeen temporally synchronized.

The vital signs storage unit 125 stores vital signs information such asbody temperature, cardioelectric potential, blood pressure, blood oxygenlevel, brainwaves, and so forth, measured by sensors attached to thebody of the patient during surgery, at the same time as the imagingperformed by the endoscope image storage unit 112. The vital signsinformation is temporally synchronized with the images stored in theendoscope image storage unit 112.

The MRI image storage unit 122 stores 3D image information includingimages of the affected area to be treated by surgery, which have beenrecorded by an MRI system before surgery.

The 3D image compositing unit 123 composites image information stored inthe MRI image storage unit 122 into images of a format which can bedisplayed on the display 190, as specified slices or a spectrogram of aspecified range.

The depth information generating unit 130 obtains left and rightdisparity of the 3D images stored in the endoscope image storage unit112, and generates depth information for the images.

The additional image position storage unit 140 stores the position wherethe vital signs information stored in the vital signs storage unit 125or the MRI image information stored in the MRI image storage unit 122 isto be displayed on the screen of the display 190.

The additional image size storage unit 150 stores the size of displayingthe additional information acquired by the additional informationacquisition unit 120 as an additional image on the screen of the display190.

The display region candidate deciding unit 160 decides candidates for adisplay region to display one or a plurality of additional informationon the screen of the display 190 as an additional image.

The depth suitability determination unit 170 detects depth differencegreater than a predetermined value, or depth contradiction, in thedifference in depth between an additional image and the main image atthe periphery of the boundary plane of the additional image displayregion. This detection is made based on information of candidates for adisplay region decided by the display region candidate deciding unit 160and depth information of the endoscope image generated by the depthinformation generating unit 130.

The suitability determination result storage unit 340 stores the valueof difference in depth between the additional image and main image whichthe depth suitability determination unit 170 has determined for eachdisplay region candidate at each predetermined time interval.

FIG. 42 illustrates an example of information stored in the suitabilitydetermination result storage unit 340. FIG. 42 illustrates thedetermination results of the depth suitability determination unit 170with regard to each additional image display region at each timeinterval.

The optimal display plan deciding unit 320 searches the informationstored in the suitability determination result storage unit 340 forresults which the has determined regarding the images stored in theendoscope image storage unit 112 at all time intervals. The optimaldisplay plan deciding unit 320 selects an optimal display region foreach time interval, from the display region candidates at each timeinterval. The optimal display plan deciding unit 320 decides theselected optimal additional image display region for each time intervalof the images stored in the endoscope image storage unit 112 to be adisplay plan. Deciding of the optimal display region is performed basedon determination standards such as additional images where the number oftimes of display position change is the least throughout all timeintervals, additional images where the amount of display position changeis the least throughout all time intervals, and so forth, for example.The display plan which the optimal display plan deciding unit 320decides is stored in the additional image display plan storage unit 330.

The image compositing unit 180 displays the 3D image stored in theendoscope image storage unit 112 on the screen of the display 190, andcomposites an image such that the vital signs information stored in thevital signs storage unit 125 and the MRI image information stored in theMRI image storage unit 122 are displayed at the additional image displayregions at each time indicated by the display plan stored in theadditional image display plan storage unit 330. The display 190 displaysthe generated image.

Operations

FIG. 43 is a flowchart illustrating operations of the 3D display device30 according to the third embodiment. FIGS. 44 and 45 are flowchartsillustrating a part of operations of the 3D display device 30 accordingto the third embodiment in detail. The operations of the 3D displaydevice 30 according to the third embodiment will be described withreference to FIGS. 41 and 43 through 45.

The 3D display device 30 first decides the time sequence of the displayregions of additional images to be displayed along with the imagesstored in the endoscope image storage unit 112 serving as the main image(step S3100). Next, the 3D display device 30 actually composites themain image and additional image, and displays the composited 3D image(step S3200).

Detailed operations of step S3100 will be described next. Referring toFIG. 44, the depth information generating unit 130 acquires an image fora unit time to be processed, from the 3D images stored in the endoscopeimage storage unit 112, ad determines whether or not there areunprocessed images remaining in the endoscope image storage unit 112(step S3110). A unit time for processing is one sample of a digitalmoving image, for example. In a case where determination is made in stepS3100 that there is an unprocessed image remaining in the endoscopeimage storage unit 112 (yes in step S3110), the flow advances to stepS1200. In a case where determination is made in step S3110 that thereare no unprocessed images remaining in the endoscope image storage unit112 (no in step S3110), the flow advances to step S3130.

The display region candidate deciding unit 160 decides in step S1200 thecandidates for the display region, from the size of the additional imagestored in the additional image size storage unit 150, and the positionof the additional image stored in the additional image position storageunit 140 (step S1200).

The depth information generating unit 130 generates depth information ofimages in processing units acquired from the endoscope image storageunit 112 (step S1300).

The depth suitability determination unit 170 compares the depth of thedisplay region obtained from the information stored in the additionalimage position storage unit 140 with the depth of the main image at theportion adjacent to the boundary line or boundary plane of theadditional image extracted in step S1300 (step S1400).

The depth suitability determination unit 170 determines whether or notthe difference in depth of the main image and additional image displayedacross the boundary line or boundary plane (step S1500). In a case wheredetermination is made in step S1500 that the difference in depth exceedsthe tolerance range, i.e., in a case where step S1500 yields “yes”, theflow advances to step S1600. In a case where determination is made instep S1500 that the difference in depth is within the tolerance range,i.e., in a case where step S1500 yields “no”, the flow advances to stepS3120.

In step S1600, the display region candidate deciding unit 160 moves thedisplay position of the additional image, and stores the moved displayposition in the additional image position storage unit 140 (step S1600).After step S1600, the flow returns to step S1200.

In step S3120, the depth suitability determination unit 170 stores, inthe suitability determination result storage unit 340, time informationof synchronizing with the main image stored in the endoscope imagestorage unit 112, region information of additional image display regioncandidates, and difference in depth between the main image andadditional image displayed across the region boundary line or boundaryplane of the additional image obtained in step S1400. The storedcontents are configured as illustrated in FIG. 42, for example. Afterstep S3120, the flow returns to step S3110.

Repeating steps S3110 through S3120 stores additional image displayregion candidates for each unit time, for all time intervals of imagesstored in the endoscope image storage unit 112.

In step S3130, the optimal display plan deciding unit 320 decides adisplay region to display an additional image for each time interval,out of the additional image display region candidates for each timeinterval stored in the suitability determination result storage unit340. The method of selecting the additional image display region may beselecting an additional image display region at each time so that thenumber of times of movement of the additional image display region as tothe main image in the overall time is the least.

Another method of selecting the additional image display region may beto select an additional image display region at each time so that thetotal amount of movement of the additional image display region as tothe main image in the overall time is the least. Optimization may alsoperformed using a standard which combines the number of times ofmovement and the distance of movement. The optimal display plan decidingunit 320 stores the additional image display region for each timeinterval that has been optimized in this way, i.e., the display plan ofthe additional images, in the additional image display plan storage unit330 (step S3140).

Performing processing of steps S3110 through S3140 decides alladditional image display regions for superimposed display on the mainimage, for all time intervals of the main images stored in the endoscopeimage storage unit 112.

Next, detailed operations in step S3200 will be described with referenceto FIG. 45. The image compositing unit 180 acquires images in it timefor display from the stereo 3D images stored in the endoscope imagestorage unit 112, and determines whether or not all images in theendoscope image storage unit 112 have been displayed (step S3210). In acase where determination is made that not all images have been displayed(no in step S3210), the flow advances to step S3220. In a case wheredetermination is made that not images have been displayed (yes in stepS3210), the flow advances to step S3250.

In step S3220 the image compositing unit 180 acquires an additionalimage display region from the additional image display plan storage unit330 corresponding to this time interval.

In step S3230 the image compositing unit 180 acquires vital signsinformation from the vital signs storage unit 125 corresponding to thistime interval. The image compositing unit 180 also follows a displaytemplate to generate an image in the additional image display regionacquired in step S3220. The display template is a template of a graph inwhich the horizontal axis represents time, for example, and the verticalaxis represents values such as body temperature, blood pressure, and soforth. The image compositing unit 180 further acquires, from the 3Dimage compositing unit 123, stereo 3D image information generated ascomputer graphics based on the information stored in the MRI imagestorage unit 122. The image compositing unit 180 generates computergraphics images within the additional image display region acquired instep S3220 (step S3230).

The image compositing unit 180 composites the additional image generatedin step S3230 with the main image acquired in step S3210, therebygenerating an entire image (step S3240).

The display 190 displays the image generated in step S3240 (step S1800).

Repeating steps S3210 through S1800 enables all additional imagescorresponding to images stored in the endoscope image storage unit 112to be displayed. Upon all images recorded in the endoscope image storageunit 112 being displayed, the operations of the 3D display device 30 end(step S3250).

Advantages and Effects

As described above, the 3D display device 30 according to the presentembodiment decides display regions for additional images when displayingadditional images over or adjacent to 3D images, such that states wherethere is a great difference in depth between within and outside theboundary of the additional image display region, or states where thereis depth contradiction are avoided. Accordingly, user discomfort andfatigue due to excessive difference in depth and depth contradiction canbe prevented. Further, the additional image display regions areoptimized over all display time intervals of the main image andadditional images, so the load on the user due to the additional imagedisplay regions frequently moving or moving great distances can beavoided, and user fatigue can be prevented.

The display region candidate deciding unit 160 has been described asdeciding the display region candidates based on information stored inthe additional image position storage unit 140 and additional image sizestorage unit 150 in the third embodiment. However, an arrangement may bemade such as in the second embodiment, and second, third, and fourthmodifications of the second embodiment, where the display-forbiddenregion storage unit 210 stores information of regions where shielding byadditional images is forbidden, and a region which would shield theaffected area which is to be treated by surgery is not set as anadditional image display region.

Also, while description is made in the third embodiment that additionalimages are automatically displayed and the user does not operate theadditional image, an arrangement may be made such as in the secondembodiment, and the second modification of the second embodiment, wherethe user inputs operations as to the additional image from the inputunit 200, so as to change the region of the additional image.

Further, while description is made in the third embodiment that theadditional image display regions are selected for each time interval,such that the number of times of movement of the total distance ofmovement of the additional image display regions as to the recoded mainimage is the least, the following processing may be performed so thatthe number of times of movement of the total distance of movement of theadditional image display regions is the least in the main image imagedin real-time as described in the first and second embodiments as well.For example, at the time of deciding an additional image display regioncandidate, the display region candidate deciding unit 160 illustrated inFIG. 1 may decides the additional image display region candidate givingpriority to a display position closest to the display region of theadditional image that was displayed immediately prior.

Fourth Embodiment

In the second embodiment, the modifications thereof, and the thirdembodiment, the main image has been described as being an image imagedby a stereo endoscope camera, and the additional images as being imagesof vital signs information temporally synchronized with the main image,and images acquired beforehand such as MRI images and so forth. On theother hand, in a 3D display device according to a fourth embodiment,images imaged by a stereo camera attached to a remotely-operated robotserve as the main image, and a map of the current position generatedfrom map information of an operating range of this robot, and currentposition information of the robot, serve as the additional images. Anoperator of the remotely operated robot monitors the 3D display device.The 3D display device 40 may display, in addition to a map of thecurrent position, an image of information of the surroundingenvironment, such as air temperature, humidity, air pressure, amount ofradiation, and so forth, in the environment in which the robot isoperating, as the additional image. In a case of displaying an image ofinformation of the surrounding environment as an additional image, thismay be displayed as a graph or the like as with the vital signsinformation images in the second embodiment.

Configuration

FIG. 46 is a block diagram illustrating a functional configuration ofthe 3D display device 40 according to the fourth embodiment. Theconfiguration in FIG. 46 is the same as the 3D display device 10according to the first embodiment illustrated in FIG. 1, other than thepoints that the main image acquisition unit 110 has been replaced by astereo camera 113, and the additional information acquisition unit 120has been replaced by a map storage unit 126, position sensor 127, and 3Dmap compositing unit 128. Other configurations are the same as with the3D display device 10 in FIG. 1. Portions which are the same as those inFIG. 1 are denoted by the same reference numerals, and description willbe omitted.

The 3D display device 40 includes the stereo camera 113, map storageunit 126, position sensor 127, 3D map compositing unit 128, depthinformation generating unit 130, additional image position storage unit140, additional image size storage unit 150, display region candidatedeciding unit 160, depth suitability determination unit 170, imagecompositing unit 180, and display 190.

The stereo camera 113 is a camera for stereo 3D imaging, attached to theremotely operated robot.

The map storage unit 126 stores map information of a range where therobot operates. The map is, for example, a blueprint of a particularbuilding, for example, and includes information of rooms on each floor.

The position sensor 127 is a sensor that measures the current positionof the remotely operated robot. A specific example is a globalpositioning satellite (GPS) system sensor and an altimeter, for example.The GPS sensor measures the lateral position, and the altimeter measuresthe floor in the building.

The 3D map compositing unit 128 composites an image indicating thecurrent position of the robot, on an image within the building displayedin 3D, based on the map information stored in the map storage unit 126and the position information of the robot measured by the positionsensor 127. The 3D map compositing unit 128 generates an image whichdisplays the structures of the building such as walls for example, in asemi-transparent form, and the robot represented by a triangular post orthe like, so that the front and back directions can be comprehended.

The depth information generating unit 130 obtains left and rightdisparity of the 3D images imaged by the stereo camera 113, andgenerates depth information for the images.

The additional image position storage unit 140 stores the position fordisplaying the images composited at the 3D map compositing unit 128 onthe screen of the display 190.

The additional image size storage unit 150 stores the size of displayingthe images composited at the 3D map compositing unit 128 on the screen.

The display region candidate deciding unit 160 decides candidates for adisplay region to display one or a plurality of additional informationon the screen as an additional image.

The depth suitability determination unit 170 detects depth differencegreater than a predetermined value, or depth contradiction, in thedifference in depth between an additional image and the main image atthe periphery of the boundary plane of the additional image displayregion. This detection is made based on information of candidates for adisplay region decided by the display region candidate deciding unit 160and depth information of the 3D image generated by the depth informationgenerating unit 130.

The image compositing unit 180 composites the 3D image imaged by thestereo camera 113 and the image composited at the 3D map compositingunit 128, so as to generated an image to be displayed on the display190. The display 190 displays the image generated by the imagecompositing unit 180.

Operations

FIG. 47 is a flowchart illustrating processing operations of the 3Ddisplay device 40 according to the fourth embodiment. FIG. 47 is thesame as FIG. 5 in the first embodiment, other than step S4100 havingbeen added to the operations illustrated in FIG. 5. The operations ofthe 3D display device 40 according to the fourth embodiment will bedescribed with reference to FIGS. 46 and 47.

First, the stereo camera 113 acquires image information for 3D displaythat has left and right disparity, and the position sensor 127 acquiresinformation of the current position of the robot (step S1100). Thecurrent position information includes lateral position and altitudeinformation.

Next, the display region candidate deciding unit 160 decides candidatesfor the display region, based on the predetermined size of theadditional image stored in the additional image size storage unit 150and the position of the additional image stored in the additional imageposition storage unit 140 (step S1200). In the fourth embodiment, oneadditional image size is stored in the additional image size storageunit 150, and one or a plurality of position information is stored inthe additional image position storage unit 140.

The depth suitability determination unit 170 extracts a boundary line orboundary plane of the display region candidate of the additional imagedecided in step S1200. A boundary plane is a depth-direction faceorthogonal to the plane of the display 190. The depth suitabilitydetermination unit 170 identifies a portion in the 3D image acquired instep S1100 in contact with the boundary line or boundary plane of theadditional image, and extracts depth information (step S1300). The depthinformation generating unit 130 generates and holds the depthinformation of the main image by the time that the main imageacquisition unit 110 acquires the main image in step S1100 and the depthsuitability determination unit 170 extracts the depth information instep S1300.

Further, the depth suitability determination unit 170 compares the depthof the display region of the additional image obtained by theinformation of the position of the additional image stored in theadditional image position storage unit 140 by the display regioncandidate deciding unit 160, with the depth of the main image at aportion in contact with the boundary line or boundary plane of theadditional image extracted in step S1300 (step S1400).

The depth suitability determination unit 170 determines whether or notthe difference in depth between the main image and additional imagedisplayed across the boundary lines or boundary plane exceeds apredetermined tolerance range (step S1500). The difference in depth is avalue obtained by subtracting the depth value of the main image nearbythe boundary line from the depth value of the additional image, and thetolerance range thereof is −1 cm to 15 cm, for example. In a case wheredetermination is made in step S1500 that the difference in depth exceedsthe tolerance range, i.e., step S1500 yields a result of “yes”, the flowadvances to step S1600. On the other hand, in a case where determinationis made in step S1500 that the difference in depth is within thetolerance range, i.e., step S1500 yields a result of “no”, the flowadvances to step S4100.

The display region candidate deciding unit 160 changes the displayposition of the additional image (step S1600). Changing of the displayposition is performed by selecting, from display positions stored in theadditional image position storage unit 140, a display position notselected as a display position candidate in step S1200. After stepS1600, the flow returns to step S1200.

In step S4100 the 3D map compositing unit 128 generates 3D computergraphics of the building, based on structure information of the buildingstored in the map storage unit 126, and renders the current position ofthe robot in the generated computer graphics of the building. The 3D mapcompositing unit 128 calculates the coordinate position corresponding tothe current position of the robot within the building, based on thelateral position and altitude of the robot acquired from the positionsensor 127, and places a symbol representing the robot at thiscoordinate position in the computer graphics of the building, such as atriangular post for example.

Thereafter, image compositing unit 180 composites the image, bydisplaying the map and the current position of the robot generated instep S4100 upon the stereo image acquired in step S1100, as the displayregion from the display region candidate decided in step S1200.

The display 190 displays the 3D image composited in step S1700 (S1800).After displaying the 3D image on the display 190 in step S1800, the flowreturns to step S1100. Repeating steps S1100 through S1800 causes the 3Ddisplay device 40 to acquire images and additional information inincrements of processing, and continue displaying of 3D images.

Advantages and Effects

As described above, the 3D display device 40 according to the presentembodiment decides display regions for additional images when displayingadditional images over or adjacent to 3D images, such that states wherethere is a great difference in depth between within and outside theboundary of the additional image display region, or states where thereis depth contradiction are avoided. Accordingly, user discomfort andfatigue due to excessive difference in depth and depth contradiction canbe prevented even when displaying 3D images of computer graphics on topof a 3D main image.

Note that the components in the above-described embodiments may each becarried out in the form of dedicated hardware, or may be carried out byexecuting a software program suitable for that component. The componentsmay be carried out by a program executing unit of a central processingunit (CPU) or some other processor reading out and executing a softwareprogram recorded in a recording medium such as a hard disk,semiconductor memory, or the like.

While a 3D display device according to one or multiple aspects has beendescribed by way of embodiments, the present disclosure is notrestricted by these embodiments. Various modifications of theembodiments and combinations of components of different embodiments maybe made by one skilled in the art without departing from the essence ofthe present disclosure, all of which may be encompassed within the scopeof one or multiple aspects. The above-described embodiments primarilycontain the disclosure according to the following aspects.

According to an aspect of the present disclosure, a three-dimensionaldisplay device includes: a display region candidate deciding unit thatdecides one candidate region from a plurality of display regioncandidates of an additional image which shields part of a main image ofa three-dimensional image on a screen; a depth suitability determinationunit that determines, in a case of assuming that the additional image isto be displayed in the candidate region which the display regioncandidate deciding unit has decided, whether or not a difference indepth between depth of the main image displayed at a boundary regionwhich is a region on the main image and within a predetermined distancefrom a boundary line of the candidate region, and the depth of theadditional image, is within a predetermined tolerance range; an imagecompositing unit that, in a case where determination is made by thedepth suitability determination unit that the difference in depth iswithin the tolerance range, superimposes the additional image upon thecandidate region on the main image, thereby compositing the main imageand the additional image, and displays an image obtained as a result ofthe compositing on the screen; and a possibly-unsuitable region decidingunit that decides, in the main image, a first region that has apossibility of the depth protruding to a near side beyond apredetermined depth range, and a second region that has a possibility ofthe depth recessing to a far side beyond a predetermined depth range.The display region candidate deciding unit further decides a candidateregion to shield the first region and the second region decided by thepossibly-unsuitable region deciding unit.

According to this aspect, in a case where the difference in depthbetween the main image and additional image across the boundary line ofthe additional image is within the tolerance range, the additional imageis displayed. Displaying the additional image in a region protrudingexcessively enables depth contradiction occurring as a result to beresolved. Accordingly, user discomfort and fatigue can be prevented.

In a case of the depth suitability determination unit determining thatthe difference in depth is not within the tolerance range, the candidateregion may be re-decided.

In this case, a candidate region where the difference in depth is withinthe tolerance range is decided. Accordingly, user discomfort and fatiguedue to excessively large difference in depth between the main image andadditional image, and depth contradiction, can be prevented.

The depth suitability determination unit may divide the boundary regioninto partial regions of a predetermined number, calculate a differencein depth between the depth of the main image displayed at the partialregion and the depth of the additional image for each partial region,and determine whether or not a maximum value of the calculateddifference in depth is within the tolerance range, thereby determiningwhether or not the difference in depth between the depth of the mainimage displayed in the boundary region and the depth of the additionalimage is within the predetermined tolerance range.

The image compositing unit may, in a case where determination is made bythe depth suitability determination unit that the difference in depth iswithin the tolerance range, composite an image where the additionalimage having a first frame is superimposed on the candidate region onthe main image, and display the composited image on the screen, and in acase where determination is made by the depth suitability determinationunit that the difference in depth is not within the tolerance range,composite an image where the additional image having a second frame,which is broader than the first frame, is superimposed on the candidateregion on the main image, and displays the composited image on thescreen.

In this case, providing a broad frame can prevent user discomfort andfatigue, even in a case where the difference in depth is great and thedifference in depth is not within the tolerance range.

The three-dimensional display device may further include adisplay-forbidden region storage unit that stores a display-forbiddenregion, which is a region on the screen where display of the additionalimage is forbidden. The display region candidate deciding unit maydecide one candidate region from the plurality of display regioncandidates of the additional image, which does not overlap thedisplay-forbidden region stored in the display-forbidden region storageunit. This prevents important portions in the main image from beingshielded from view by the additional image.

The depth suitability determination unit may adds a correction value,which is larger the closer a distance from the candidate region to thedisplay-forbidden region is, to the difference in depth, and thereupondetermine whether or not the difference in depth is within the tolerancerange.

In this case, candidate regions which are farther from thedisplay-forbidden region are more readily selected even if thedifference in depth thereof is the same.

The shorter a distance between a camera which has imaged the main imageand an object displayed in the display-forbidden region is, the largerthe size of the display-forbidden region may be.

In this case, the closer the distance is, the larger the object isdisplayed in the screen. Accordingly, the size of the display-forbiddenregion is changed in accordance with the size of the object.

The three-dimensional display device may further include apossibly-unsuitable region storage unit that writes the first region andthe second region to the possibly-unsuitable region storage unit as apossibly-unsuitable region.

The display region candidate deciding unit may decide the candidateregion giving priority to a candidate at a position closest to thedisplay region of the additional image displayed on the screenimmediately prior. In this case, the position of the display region ofthe additional image can be prevented from greatly changing.

The three-dimensional display device may further include an optimaldisplay plan deciding unit that decides a candidate region atpredetermined time intervals, from candidate regions regarding which thedifference in depth is within the tolerance range according to thedetermination results of the depth suitability determination unitregarding whether or not the difference in depth between the depth ofthe main image and the depth of the additional image is within thetolerance range, for each of the plurality of candidates of the displayregion of the additional image, at every predetermined time interval, sothat from the start of displaying the main image, which is a prerecordedthree-dimensional image, to ending of displaying, the distance ofmovement or number of times of movement of the display region of theadditional image is smallest. The image compositing unit may compositean image where the additional image is superimposed on the main image,such that the additional image is displayed at the candidate region foreach predetermined time interval decided by the optimal display plandeciding unit, and display the composited image on the screen.

In this case, the position of the display region of the additional imagecan be prevented from greatly changing when playing a recorded mainimage.

The main image may be a three-dimensional image imaged by an endoscopecamera. In this case, the 3D display device can be used for endoscopesurgery.

The three-dimensional display device may further include adisplay-forbidden region storage unit that stores a display-forbiddenregion, which is a region on the screen where display of the additionalimage is forbidden. The display region candidate deciding unit maydecide one candidate region from the plurality of display regioncandidates of the additional image, which does not overlap thedisplay-forbidden region stored in the display-forbidden region storageunit, and the display-forbidden region may be a region of the main imageincluding an image of an affected area to be treated by surgery.

In this case, a situation can be prevented where the image of theaffected area is hidden by the additional image, impeding the surgery.

The additional image may be an image indicating at least one of bloodpressure, blood oxygen level distribution, respiration, expiration, bodytemperature, cardioelectric potential, brainwaves, and pulse waves, ofthe patient during surgery.

In this case, an additional image indicating additional informationnecessary for surgery can be displayed on the screen, thus supportingthe surgeon in performing the surgery.

The three-dimensional display device may further include an input unitthat accepts instruction of at least one of a position and a size of thedisplay region of the additional image displayed on the screen. Thedisplay region candidate deciding unit may decide a candidate region forthe display region of the additional image, based on the instructionwhich the input unit has accepted.

In this case, an additional image is displayed in a case where thedifference in depth between the main image and additional image across aboundary line of the additional image is within the tolerance range.Accordingly, user discomfort and fatigue due to excessively largedifference in depth between the main image and additional image, anddepth contradiction, can be prevented at the time of displaying anadditional image in a display region having a position or size specifiedby the user.

The input unit may further accept instruction to change the size of thedisplay region, the display region candidate deciding unit changing thesize of the candidate region of the display region following theinstruction to change the size which the input unit has accepted. Inthis case, the size of the display region of the additional image cab bechanged.

The main image may be a video inside the body of the patient. In thiscase, the 3D display device can be used for surgery.

An image of a surgical instrument may be displayed in the main image.The input unit may accept input of the position, using information ofthe position of the surgical instrument in the main image.

In this case, the user can specify the size of the additional imageusing the instruments being used for surgery, without using any specialinterface device. That is to say, the user who is the surgeon caninstruct the size of the additional image without releasing the surgicalinstruments. Thus, the 3D display device can be operated withoutsacrificing efficiency in surgery.

The input unit may include a passage detecting unit that detects whetheror not a tip of the surgical instrument in the main image has passedthrough the display region of the additional image, and a displaycontrol signal generating unit that, in a case where the passagedetecting unit has detected the passage, changes the size of thecandidate region of the display region in accordance with the directionof passage of the surgical instrument.

In this case, the size of the candidate region can be changed so thatthe tip of the surgical instrument is not hidden by the additionalimage. This can prevent the tip of the surgical instrument being hiddenby the additional image and impeding surgery.

The input unit may further accept instruction to change the position ofthe display region, the display region candidate deciding unit changingthe position of the candidate region of the display region following theinstruction to change the position which the input unit has accepted. Inthis case, the position of the display region of the additional imagecan be changed.

The three-dimensional display device may further include adisplay-forbidden region storage unit that stores a display-forbiddenregion, which is a region on the screen where display of the additionalimage is forbidden. The display region candidate deciding unit maydecide the candidate region, which does not overlap thedisplay-forbidden region stored in the display-forbidden region storageunit, based on the instruction which the input unit has accepted. Thisprevents important portions in the main image from being shielded fromview by the additional image.

The main image may be a video inside the body of the patient, and theadditional image may be an image of graphs of a plurality of types ofbiological data of the patient, which changes over time.

In this case, an additional image of graphs of multiple types ofbiological data necessary for surgery can be displayed on the screen,thus supporting the surgery being performed by the surgeon.

The display region candidate deciding unit may decide the candidateregion having a shape corresponding to a layout of the graphs of theplurality of types of biological data. In this case, the display regionof the additional image can be changed according to the layout of thegraphs.

The layout may include a layout where the graphs of the plurality oftypes of biological data are arrayed vertically, and a layout where thegraphs of the plurality of types of biological data are arrayedhorizontally.

In a case where the depth suitability determination unit determines thatthe difference in depth of the candidate region is not within thetolerance range even if the position or size is changed, the displayregion candidate deciding unit may change the layout of the graphs ofthe plurality of types of biological data.

The display region candidate deciding unit may extract a portion in themain image where the depth exceeds a predetermined value as being aportion where protrusion from the screen is great, and decide acandidate region of the display region of the additional image to shieldthe portion where protrusion from the screen is great.

In this case, the additional image can be displayed to shield a portionprotruding to the near side excessively, for example, and accordinglydepth contradiction between the main image and additional image, andexcessively large difference in depth, can be prevented.

Software which realizes the 3D display device according to theembodiments, and so forth, is a program such as follows. The programcauses a computer to execute first deciding of one candidate region froma plurality of display region candidates of an additional image whichshields part of a main image of a three-dimensional image on a screen;determining, in a case of assuming that the additional image is to bedisplayed in the candidate region which the display region candidatedeciding unit has decided, whether or not a difference in depth betweendepth of the main image displayed at a boundary region which is a regionon the main image and within a predetermined distance from a boundaryline of the candidate region, and the depth of the additional image, iswithin a predetermined tolerance range; superimposing, in a case wheredetermination is made in the determining that the difference in depth iswithin the tolerance range, the additional image upon the candidateregion on the main image, thereby compositing the main image and theadditional image, and displaying an image obtained as a result of thecompositing on the screen; and second deciding, in the main image, of afirst region that has a possibility of the depth protruding to a nearside beyond a predetermined depth range, and a second region that has apossibility of the depth recessing to a far side beyond a predetermineddepth range. The first deciding further decides a candidate region toshield the first region and the second region decided in the seconddeciding.

The present disclosure is applicable to a display device which displays3D images or 3D video, and particularly to a display device of imagesand the like imaged by a stereo endoscope camera, a display device ofrecorded contents obtained by recording images, a medical 3D imagedisplay device, a monitor display device for are remotely operatedrobot, and so forth.

What is claimed is:
 1. A three-dimensional display device configured todisplay a main image and an additional image on a screen, thethree-dimensional display device comprising: a memory that storesinstructions; and a processor, when executing the instructions stored inthe memory, that performs operations including: deciding one candidateregion from a plurality of region candidates for the additional image tobe superimposed on the main image on the screen; determining, a based ona boundary region that is within a predetermined distance from aboundary line between the candidate region and outside of the candidateregion in the main image, whether a difference between a depth of themain image displayed at the boundary region and a depth of theadditional image is within a predetermined tolerance range;superimposing, when it is determined that the difference in depthbetween the depth of the main image displayed at the boundary region andthe depth of the additional image is within the predetermined tolerancerange, the additional image upon the main image at the candidate region,thereby composing a composite image of the main image and the additionalimage; and displaying the composite image on the screen.
 2. Thethree-dimensional display device according to claim 1, wherein, when itis determined that the difference in depth is not within thepredetermined tolerance range, the processor re-decides the candidateregion.
 3. The three-dimensional display device according to claim 1,wherein the processor divides the boundary region into a predeterminednumber of partial regions, calculates a difference in depth between thedepth of the main image displayed at the partial region and the depth ofthe additional image for each partial region, and determines whether ornot a maximum value of the calculated difference in depth is within thepredetermined tolerance range, thereby determining whether or not thedifference in depth between the depth of the main image displayed in theboundary region and the depth of the additional image is within thepredetermined tolerance range.
 4. The three-dimensional display deviceaccording to claim 1, wherein the processor, when it is determined thatthe difference in depth is within the predetermined tolerance range,composes, as the composite image, an image where the additional imagehaving a first frame is superimposed on the candidate region on the mainimage, and displays the composite image on the screen, and when it isdetermined that the difference in depth is not within the predeterminedtolerance range, composes, as the composite image, an image where theadditional image having a second frame, which is broader than the firstframe, is superimposed on the candidate region on the main image, anddisplays the composite image on the screen.
 5. The three-dimensionaldisplay device according to claim 1, further comprising: a storage thatstores a display-forbidden region, which is a region on the screen wheredisplay of the additional image is forbidden, wherein the processordecides one candidate region from the plurality of region candidates forthe additional image, which does not overlap the display-forbiddenregion.
 6. The three-dimensional display device according to claim 5,wherein the processor adds a correction value, which is larger thecloser a distance from the candidate region to the display-forbiddenregion is, to the difference in depth, and thereupon determines whetheror not the difference in depth is within the predetermined tolerancerange.
 7. The three-dimensional display device according to claim 5,wherein, the shorter a distance between a camera which has imaged themain image and an object displayed in the display-forbidden region, thelarger the size of the display-forbidden region.
 8. Thethree-dimensional display device according to claim 1, wherein theprocessor decides the candidate region giving priority to a candidate ata position closest to the display region of the additional imageimmediately previously displayed on the screen.
 9. The three-dimensionaldisplay device according to claim 1, wherein the processor, whenexecuting the instructions stored in the memory, further performsoperations including: deciding a candidate region at predetermined timeintervals, from candidate regions in which the difference in depth iswithin the predetermined tolerance range according to the determinationresults regarding whether or not the difference in depth between thedepth of the main image and the depth of the additional image is withinthe predetermined tolerance range, for each of the plurality ofcandidates of the display region of the additional image, at everypredetermined time interval, so that from a start of displaying the mainimage, which is a prerecorded three-dimensional image, to an end ofdisplaying, the distance of movement or number of times of movement ofthe display region of the additional image is smallest, wherein theprocessor composes, as the composite image, an image where theadditional image is superimposed on the main image, such that theadditional image is displayed at the candidate region decided for eachpredetermined time interval, and displays the composite image on thescreen.
 10. The three-dimensional display device according to claim 9,wherein the main image is a three-dimensional image imaged by anendoscope camera.
 11. The three-dimensional display device according toclaim 10, further comprising: a storage that stores a display-forbiddenregion, which is a region on the screen where display of the additionalimage is forbidden, wherein the processor decides one candidate regionfrom the plurality of region candidates of the additional image, whichdoes not overlap the display-forbidden region stored in the storage, andwherein the display-forbidden region is a region of the main imageincluding an image of an affected area to be treated by surgery.
 12. Thethree-dimensional display device according to claim 1, wherein theadditional image is an image indicating at least one of blood pressure,blood oxygen level distribution, respiration, expiration, bodytemperature, cardioelectric potential, brainwaves, and pulse waves, ofthe patient during surgery.
 13. The three-dimensional display deviceaccording to claim 1, further comprising: an input that acceptsinstruction of at least one of a position and a size of the displayregion of the additional image displayed on the screen, wherein theprocessor decides a candidate region for the display region of theadditional image, based on the instruction which the input has accepted.14. The three-dimensional display device according to claim 13, whereinthe input further accepts instruction to change the size of the displayregion, and wherein the processor changes the size of the candidateregion of the display region following the instruction to change thesize which the input has accepted.
 15. The three-dimensional displaydevice according to claim 13, wherein the main image is a video of aninside of a body of a patient.
 16. The three-dimensional display deviceaccording to claim 15, wherein an image of a surgical instrument isdisplayed in the main image, and wherein the input accepts input of theposition, using information of the position of the surgical instrumentin the main image.
 17. The three-dimensional display device according toclaim 16, wherein the input includes a passage detector that detectswhether or not a tip of the surgical instrument in the main image haspassed through the display region of the additional image, and a displaycontrol signal generator that, when the passage detector has detectedthe passage, changes a size of the candidate region of the displayregion in accordance with a direction of passage of the surgicalinstrument.
 18. The three-dimensional display device according to claim13, wherein the input further accepts instruction to change the positionof the display region, and wherein the processor changes the position ofthe candidate region of the display region following the instruction tochange the position which the input has accepted.
 19. Thethree-dimensional display device according to claim 13, furthercomprising: a storage that stores a display-forbidden region, which is aregion on the screen where display of the additional image is forbidden,wherein the processor decides the candidate region, which does notoverlap the display-forbidden region, based on the instruction which theinput has accepted.
 20. The three-dimensional display device accordingto claim 15, wherein the main image is a video of an inside of a body ofa patient, and wherein the additional image is an image of graphs of aplurality of types of biological data of the patient, which changes overtime.
 21. The three-dimensional display device according to claim 20,wherein the processor decides the candidate region having a shapecorresponding to a layout of the graphs of the plurality of types ofbiological data.
 22. The three-dimensional display device according toclaim 21, wherein the layout includes a layout where the graphs of theplurality of types of biological data are arrayed vertically, and alayout where the graphs of the plurality of types of biological data arearrayed horizontally.
 23. The three-dimensional display device accordingto claim 21, wherein, when the processor determines that the differencein depth of the candidate region is not within the predeterminedtolerance range even if the position or size is changed, the processorchanges the layout of the graphs of the plurality of types of biologicaldata.
 24. The three-dimensional display device according to claim 13,wherein the processor extracts a portion in the main image where thedepth exceeds a predetermined value as being a portion where protrusionfrom the screen is great, and decides a candidate region of the displayregion of the additional image to shield the portion where protrusionfrom the screen is great.
 25. A three-dimensional display method fordisplaying a main image and an additional image on a screen of athree-dimensional display device, the method comprising: deciding onecandidate region from a plurality of region candidates for theadditional image to be superimposed on the main image on the screen;determining, based on a boundary region that is within a predetermineddistance from a boundary line between the candidate region and outsideof the candidate region in the main image, whether a difference betweena depth of the main image displayed at the boundary region and a depthof the additional image is within a predetermined tolerance range; andsuperimposing, when it is determined that the difference in depthbetween the depth of the main image displayed at the boundary region andthe depth of the additional image is within the predetermined tolerancerange, the additional image on the main image, thereby composing, as acomposite image, the main image and the additional image, and displayingthe composite image on the screen.
 26. A non-transitorycomputer-readable recording medium storing a program causing a computerto execute the three-dimensional display method according to claim 25.27. The three-dimensional display device according to claim 1, whereinthe processor determines whether the difference obtained by subtractingthe depth of the main image displayed at the boundary region from thedepth of the additional image is equal to or greater than apredetermined tolerance value, and wherein the processor superimposesthe additional image upon the main image at the candidate region, whenit is determined that the obtained difference is equal to or greaterthan the predetermined tolerance value.
 28. The three-dimensionaldisplay device according to claim 1, wherein the processor determineswhether the main image displayed at the boundary region protrudes to anearer side of the screen than the additional image, and wherein theprocessor superimposes the additional image upon the main image at thecandidate region, when it is determined that the main image displayed atthe boundary region does not protrude to the nearer side of the screenthan the additional image.